|
I think there are a couple of great points here, and I'll try to break things down and address each point independently. 1. re: "real" user reviews — right now, we do do this, just in a different way vs. what I think you're thinking of. For all of our guides so far, we've conducted a lot of interviews with founders who've used each product. However, we ultimately synthesize this data, pair it up with our own findings from our testing, and draw our own conclusions. We think this is close to the optimal way of doing things. We found that sites which rely 100% on user reviews usually hit a ceiling in terms of usefulness, as the reviews are typically on the shallower side, which compounds b/c high-level reviews typically end up talking about the same things, so they're often redundant as well. By getting founder feedback, we're able to get a lot of datapoints to guide our findings and to surface less common issues, but by testing and ultimately writing these guides ourselves, we're able to both spend the time to elaborate on important aspects in depth (e.g. caveats, background info, etc.), while also being able to present info from a bigger picture perspective (e.g. this is how X, Y, and Z compare, etc.). The main downside is that it requires a lot of additional work. 2. re: Gartner + incentives — I believe that you identified the problem correctly, but misattributed the underlying cause. Gartner's incentives are misaligned b/c vendors who pay have an edge up on those who don't, for reasons you correctly identified. Importantly, this is not isolated to Gartner — in fact, the incentives are just as misaligned for a lot of the 100% UGC review sites. For those sites, if a vendor pays, they're allowed to cherrypick their own users to invite for an "organic review." If the vendor doesn't pay, then their rating will be relatively lower b/c it becomes much harder for them to get their best users to write a review on the site. So in general, the problem isn't caused by whether you're a UGC review site or write guides yourself — it's caused by, well, whether or not you're doing something slimy to make money. Edit: Oh, and I just saw that you mentioned Capiche at the end. I just want to say that none of this comment is directed at them — I sincerely think that they're passionate, motivated founders who deeply care about doing good for the startup community. |