[inteoryx]

An earlier idea I had for this was to put two tweets side by side and ask the user to say which tweet was more offensive. Then, I figured I could use the ELO rating system to come up with an "Offensive score" for each tweet. e.g. "This tweet has an offensiveness of 2200" or something like that. I could then compare the average offensiveness of tweets that Twitter considered offensive versus not.

I wound up not going with that approach because many times you just have two completely innocuous tweets and picking which of the two of them is "more offensive" is just arbitrary. I could have curated the tweets so that only ones that were kind of offensive were in the quiz, but then I might be putting my thumb on the scales to get the answer I already believed in. I'd also need to get lots of ratings for each Tweet to have a stable score.

I think your idea is pretty interesting. It would allow a conclusion like "The average tweet Twitter marks as offensive is X% likely to offend a rater." I was coming at it more from a "Twitter's offensive identification is like random chance" perspective rather than just trying to assess quality. If I had considered this idea while creating the quiz I might have gone with it!