Famously, John Quincy Adams served in the House for nine terms (17 years) after his single term as president, where he spent most of his time fighting to abolish slavery.
Andrew Johnson won election to the Senate shortly before his death, in 1875. This was after several attempts to win political office in his post-Presidency years had failed.
Reportedly, he hated being President and wished he'd never done it, but loved being a judge.