If he doesn’t walk out during the current series, it looks like Gary Barlow will hang up his X Factor hat when it finishes.

The singer-turned-judge is reportedly planning to quit the talent show in order to focus on Take That’s next album and tour.

He is also said to be embarrassed by the antics of the other judges on Sunday night when his act Carolynne Poole lost out to Rylan Clarke.

Gary walked off the set at the weekend after his act was kicked off the show.

The incident is said to have created a rift between Gary and fellow judge Louis Walsh who sent the vote to deadlock despite saying that he wanted to save Carolynne.

Apparently the two hardly uttered a word to each other when the entire team gathered for crisis talks at Mayfair’s Dover Street Arts Club earlier this week.

Speaking about the whole thing, an insider told The Sun newspaper: “The events of the last week have really confirmed in Gary’s mind that he needs to get back to what he loves best.”

They added: “Gary is going to give this series his all and then it will be full steam ahead for Take That.”

Gary met up with the rest of the band to discuss their next step.

All of the boys, including Robbie who left the band for the second time two years ago, met up recently to discuss the band’s future plans. Rumour has it they are hitting the studio by the end of the year and are planning a huge tour for 2014.