You don't have to contribute GPL changes back. Whomever you give a binary to you'll have to give the source to if they ask. It's pretty much that simple.
That's a distinction without a difference as anyone who receives the binary must also have access to the source and be free to redistribute that source, so effectively all derivative works of GPL'd code are GPL'd unless you refuse to distribute.