With over half of the clubs in the Premier League and Championship promoting gambling firms on their shirts, Sweet admitted he had been approached to do something similar at Kenilworth Road.
However, they have turned down such shirt and stadium sponsorship offers, despite the obvious lucrative reward.
Speaking to BBC Look East, Sweet said: “We don't want to promote excessive gambling behaviour through our support base and our players.
“I don't think I'm going to be fired because I'm turning down that kind of income from our board, because together we all think and believe the same things as part of our principles.
Luton council slammed for new ban on night-time deliveries that will 'cause havoc'
Have your say over the future of Stockwood Park in Luton
Charles Bronson - Luton's most notorious prisoner - hopes to be out of jail in time for Christmas
Food hygiene standards in Luton are declining says council
Luton will not have hosepipe ban as southeast England officially in drought
“We don't feel comfortable with advertising or carrying a sponsorship of a gambling brand unless we are forced to do so obviously.
“Over the years we probably get an offer every year from at least one gambling firm.
“I would put it - certainly over the years of our custodianship - of well in excess of half a million pounds lost probably, at a guess, but we're very comfortable with that."