The 56-year-old had moved to Kenilworth Road as number two to Nathan Jones in October 2018 after Paul Hart left the club for Notts County.
He carried on when Jones also departed and Mick Harford took charge in January, as the Hatters won promotion to the Championship.
With the club then appointing Graeme Jones as their new manager, Rutter was offered the chance to stay on, but in a different role.
He eventually turned it down, opting to move to Greek club Atromitos, where he was named assistant coach, and on his decision, Rutter said: “Yes it was (a difficult one), but everything’s about opportunities.
“When I came to Luton with Nathan, it was a great opportunity for me, because I’d never worked in English football before at a professional level.
“I’d worked in the Conference, but not in the Football League.
“But I was still living in Yeovil and that was commuting a couple of times a week from Yeovil and a couple of times a week from mum’s in Northampton, and to be fair, it was just so intense, there was just no respite.
“Last season was a fantastic season, I was so pleased for Mick as well and everybody, it was great.
“It’s a superb club the way they run it, it really is.
“Sometimes you go to places and they’ve got a profile and when you get inside, you realise it’s completely different, but their profile about integrity and being ethical, they’ve just been superb.
“I was delighted to finish the season the way we did and win it, it was great for Mick, great for the club and ultimately it made it difficult to move on.
“But from a personal point of view, it was difficult for me with all the travelling.
“Then obviously Graeme’s come in and I got on well with Graeme, he’s a good lad, but he wanted to bring a couple of lads with him and my role would ultimately have changed to a considerable degree.
“It would have been less compliant with the travelling and the requirements, it would have been a bit of this and a bit of that.
“So ultimately I had to make a decision and as counter-intuitive as it sounds, going abroad means I get more time with my family.”
When asked whether he would have stayed had he remained as assistant to Jones, Rutter continued: “I totally respect that when he came in, he needs to have people with him that he’s comfortable with.
“Although we get on okay, everyone has working practices and there’s certain people they know they gel with, so there’s absolutely no issue with that at all.
“I spoke to Gary (Sweet, chief executive) at length about it, I spoke to Graeme at length about.
“They were very accommodating about it, they tried very hard to find something that would suit, but it was just ultimately, when I looked at it all, the logistics of it made it very difficult.
“I can’t praise them enoughthough, they’ve been absolutely superb.
“With the Europa League qualifiers, we (Atromitos) start back eight days earlier, so there was a bit of timing on it.
“I didn’t want to get far into the discussions with someone before I had got an agreement with Luton that they were happy with me to go and I was happy to go.
“Ultimately, I think it will suit me certainly from a personal and professional point of view and hopefully it will suit Luton as well.
“I was there for a purpose and that purpose was achieved and the club will go from strength to strength, I’m absolutely sure, as they’ve got a great group of people behind the scenes.
“It’s a fabulous place to be and all you can do really is wish them all.”
Being based in the West country, Rutter revealed just how much a toil journey down to Bedfordshire had been for him at times.
He said: “If you took an average week, if we had a game at Luton at home on Saturday, by the time you get out of the stadium, it’s half past six, seven o’clock.
“I get back to Yeovil at 10, if we’re in on the Monday, then I’ve spent three hours on the Sunday editing the game.
“So I go to bed when I get home, I get up to have my breakfast, I spend three hours doing that, I get a couple of hours in the afternoon with the family, then I’m back on the road at 5 o’clock in the morning on Monday.
“That happened week after week, in fact I didn’t get home at all for the last three and a half weeks of the season.
“When you’re abroad the season’s are less intensive as there’s less teams in the league, you get a Christmas break, and you get just more time.
“When the family come here, they come for a holiday, I train, then I go home, so I might go out every day, but I’m out for three or four hours and then I’m home and my family are there.
“Whereas I’d go home in Luton, I’d go up to my mum’s, which was great, but I wasn’t seeing my family.
“So in a strange sort of way, you get more time and better quality time with your family as there’s less intensity in the league.
“It was another experience though, a fantastic year, unbelievable achievement, great to have worked with some, and they were, truly really wonderful people.
“They’re a pleasure to be involved with and all I can do is wish them well next season.”