Then you have to worry about security from side channel attacks. Even things like the iOS or Android task-switcher UI caching screenshots of your app can be vectors.
Oh yeah, making an app that is easy to use and still secure is definitely not easy. But at least you would know how you implemented the encryption on it.