https://hackaday.io/project/19035-zerophone-a-raspberry-pi-s...
https://openalia.wordpress.com/2012/06/09/diy-totally-open-s...
...then start deleting code out of it, hardening it, and using memory-safe apps. A nice step on the way to a truly, dumb phone that might take a lot of custom work.