Demolition work on the former High Town Recreation Centre in Old Bedford Road will start in the next few weeks.
Housing will be built on the site and it is expected it will hold between 60 and 80 new homes.
The building has stood derelict for a number of years, making it an eyesore for the local area.
Mo Harkin, Head of Housing, said: “As the building has been rusting away for a number of years I know local people will be pleased to hear that the demolition process has started. Residents told us they wanted to see the site used for affordable housing so it is exciting that the process is underway. The new homes will allow vulnerable local families in desperate housing need to move into affordable homes.”
Once the site is clear more details will be announced about the specific plans for the site and how it will help meet Luton’s growing housing need.
The demolition is expected to be finished in December.