The hosts had taken the lead moments after half time as Kev Kilroy’s free kick sailed into the net.
However, Arlesey were level on 57 minutes through Lewis Wilson.
They then snatched victory with three minutes to go as Wilson set up Christian Tavernier to score the winner.
Rovers are at home to AFC Rushden & Diamonds this weekend.
Crawley Green went through in the SSML Division one Cup, winning 2-1 at Winslow United.
After going 1-0 down, Crawley replied with goals from Sam Halfpenny and JJ Billy to ensure they progressed.
The 61 FC (Luton) were hammered 12-1 by St Albans City Reserves in their League Challenge Trophy tie as Bradley Currington netted a consolation.