Passenger who got to hold the stick for a bit. Flying the plane is easy. Flying it roughly at some altitude in a rough direction you're given is also easy.
Now, landing it? That's an entirely different beast for sure.
It is, but it's not as difficult as you think. PPL students are landing the plane solo within 10-30 hours of flight time, during which they will have landed maybe 40 times.