[Lewisham]

The reason I've been reading it all is because I'm writing my thesis on it :)

I haven't spent too much time with Khan Academy, so can't say too much about it. There are two things to keep in mind:

1. Extrinsic motivation erodes intrinsic motivation. The most obvious example is asking a kid to take out the trash. Once you pay him/her, they'll only do it for money again. While this example is silly because no-one has any intrinsic motivation to take out the trash, it gets really dicey when you add rewards to things like reading. Are they reading to develop a love of reading, or for the carrot you put at the end? What happens when the carrot goes away? The research suggests that once the carrot disappears, so does the reading. The psychology isn't completely cut-and-dried on this, as no psychology seems to be, but it's compelling to me, at least.

The problem is that some scholars, particularly Alfie Kohn, would say that things like gold stars are rewards, and thus you're eroding things that way. However, you have to balance that with providing legitimate feedback for knowing how well you're doing, as people want to feel competent. Both these come from self-determination theory (read up on it, it's neat). The trick is to make sure the reward doesn't try to control behavior, but rather provide feedback on behavior. eg. "you did x 5 times" (good) vs "you did x 5 times as required" (bad). It's a really fine line.

As far as I can see, the main goals in Khan Academy seem to be informational and are good.

2. Badges are game systems themselves (I heard this from Nicole Lazzaro [1]), and they will result in increasing behavior of certain things for people that pay attention. They do not necessarily result in increased behavior of value. eg. a badge for posting 5 times is earned whether your posts are insightful, or whether they simply say "first post". And, again, these are extrinsic rewards, and once they dry up, people stop caring. There's also a strong possibility that rewards suffer from inflation, and you have to hit people with bigger and bigger rewards once they start relying on them.

The badges at Khan Academy make me wary [2]. Check out this one: "Quickly & correctly answer five questions in a row." What's the goal here? Khan Academy's goal, one would think, is to encourage students to think deeply about problems and arrive to correct solutions. It spends a lot of time saying students are able to work at their own pace, and this is a benefit, but then we have a badge which encourages quick, possibly slap-dash, behavior. It's more controlling than informational, requesting the student do something not necessarily natural to him/her in order to achieve it.

The ones given out for achieving proficiency are better; they align with what they want students to do and provide informational feedback that students have done it, increasing their motivation.

====

Now, this is not to say that all achievement systems are bad or doesn't work. Stack Overflow's seems to do really well. If it was doing poorly, we'd expect a large drop off of people when the extrinsic rewards dry up, largely at reputation 10,000, when you get moderator tools. The current data from Stack Overflow indicates only about 5% or so of users drop off once they get the tools (I can dig out references if you're interested). Upvotes and achievements provide useful feedback about your mastery of SO, so with that low attrition rate, it indicates they are doing things right.

The elephant in the room that Stack Overflow presents is that it may indicate that gamification does work in its entirety, and that all this motivation theory is invalid in this domain for some reason. This is possible, but such a strong statement would require much harder evidence than we have. Right now, it's more plausible that Stack Overflow is simply doing things well and seems to be aligning with the recommendations of the research, rather than indicating a wider problem with the applicability of this stuff.

[1] http://www.xeodesign.com/founder.html [2] http://www.khanacademy.org/badges

[Estragon]

For me, Stack Overflow's most rewarding feedback is social recognition for having been helpful, which is largely independent of the formal rewards it offers. I bet it's the same for a lot of other people too.

[Lewisham]

Yes, pretty much. The other open question is what Stack Overflow would be if they removed the reputation system entirely.

The literature on motivation from people that subscribe to self-determination theory indicates I think they would mostly prefer the system gone. Any reward system you put in place is bad, it's just there are certain degrees of bad.

My gut tells me that Stack Overflow has benefitted from the reputation system as another explicit feedback on how well you are doing. I think the badges are largely unimportant, save for being a more fun way of saying "here's what you can do on the site" than another tedious FAQ. If the badges were important, then we head down a nasty extrinsic motivation road. I think it's done better with the rep system than without.

That said, I sometimes wonder how many people are put off by the built-in competitive aspect of Stack Overflow; any writing you do is evaluated and measured against others. For competitive people that end up bumped down all the time, that seems like it would be disheartening, for social people always been graded it might be disheartening too. The winner-takes-all set up, rather than, say, a wiki answer that people are constantly tweaking, seems like it could be problematic. I know that I stopped bothering answering questions very quickly as I felt I had no mastery of the art, other people were getting the upvotes, so I felt like my answers were largely ignored.

[jasonrr]

Loving the thoughtfulness here, and I have a few more resources I need to read now.

Re: Quick and Correct badges -

It is important in many cases to be able to recall math facts quickly. Automatic recall has also been correlated with understanding of more difficult concepts (for example: [1]). I am by no means saying that we've gotten the reward system perfectly right, but we were thoughtful when creating it.

[1] http://www.ldonline.org/spearswerling/Developing_Automatic_R...

[Lewisham]

I certainly did not mean to indicate I didn't think you guys had not put a lot of thought into things. If anything, my point was that all of these things are really hard, and I certainly don't count myself among the handful of people in the world who I would trust to get it right. I also have zero pedagogical chops, so don't take my word for anything!

My gut says that, if asked, I would probably recommend that the automatic recall goal is better placed as part of individual, fun "quickfire" lessons "Get these five right before the timer runs out". This would help to ensure you limit the behavior to a place where you want it, rather than somewhere where you might prefer the students to slow down and absorb what they're doing.

You could offer an informational badge showing the student completed a quickfire lesson, and I think you'd then be closer to kosher on my read of the psych literature.

But like I said, don't take my word for it :)

[oskarth]

Thank you for your in depth and insightful reply - what you say makes a lot of sense. Good luck with your thesis and I would personally love to see a summary of your findings posted here on HN :)