[ijpsud]

Have you thought about collecting reviews from "real" users of the products to supplement your analysis?

There's potentially a misaligned-incentives problem with review sites which themselves do the reviews. The latest episode of the FYI podcast[0] goes through this problem with the founder of Capiche[1] (looks like a competitor of yours) and the founder was explaining how firms like Gartner basically have two sets of customers: The software buys and the software sellers. The software sellers can actually pay Gartner to get in front of an analyst and sell them on their product.

I don't know to what extent this makes Gartner's recommendations less useful, but it seems like it would make it hard for smaller companies to get on the radar. I think the fundamental problem is that the analysts are not actual users of the software.

So I like Capiche's direction, but I dislike the fact that they're a "closed" invite-only website.

[0] https://twitter.com/ARKInvest/status/1261421627717218305

[1] https://capiche.com

[awwstn]

Thanks for mentioning Capiche! I think there's so much opportunity in this space so I'm stoked to see anyone else joining in.

At the very start, I wanted Capiche to be essentially Wirecutter for SaaS. We ended up going in a different direction, but I'm super impressed with Satchel's execution here, and as someone who spends a crazy amount of time on this stuff, I learned a bunch of new things reading through the site today.

I really don't want to hijack @fission's launch discussion, but thought I'd just reply to your comment on the invite-only thing: our site will be wide open soon-ish. Just taking our time building a community around high quality discussion (most of which has little to do with comparisons/reviews), but eventually will open it all up. Think of it like Stack Overflow's private beta...just a step in the process.

[Fission]

Hey Austin, I appreciate the thoughtfulness and the kind words.

I think even though we're working in different directions, we share a lot of the same values and see similar issues with the status quo. When (I hope not if) quarantine's over, let's grab a coffee :) — my email's in my profile

[1cvmask]

Any chance we can join now?

[Fission]

I think there are a couple of great points here, and I'll try to break things down and address each point independently.

1. re: "real" user reviews — right now, we do do this, just in a different way vs. what I think you're thinking of. For all of our guides so far, we've conducted a lot of interviews with founders who've used each product. However, we ultimately synthesize this data, pair it up with our own findings from our testing, and draw our own conclusions.

We think this is close to the optimal way of doing things. We found that sites which rely 100% on user reviews usually hit a ceiling in terms of usefulness, as the reviews are typically on the shallower side, which compounds b/c high-level reviews typically end up talking about the same things, so they're often redundant as well. By getting founder feedback, we're able to get a lot of datapoints to guide our findings and to surface less common issues, but by testing and ultimately writing these guides ourselves, we're able to both spend the time to elaborate on important aspects in depth (e.g. caveats, background info, etc.), while also being able to present info from a bigger picture perspective (e.g. this is how X, Y, and Z compare, etc.). The main downside is that it requires a lot of additional work.

2. re: Gartner + incentives — I believe that you identified the problem correctly, but misattributed the underlying cause. Gartner's incentives are misaligned b/c vendors who pay have an edge up on those who don't, for reasons you correctly identified. Importantly, this is not isolated to Gartner — in fact, the incentives are just as misaligned for a lot of the 100% UGC review sites. For those sites, if a vendor pays, they're allowed to cherrypick their own users to invite for an "organic review." If the vendor doesn't pay, then their rating will be relatively lower b/c it becomes much harder for them to get their best users to write a review on the site.

So in general, the problem isn't caused by whether you're a UGC review site or write guides yourself — it's caused by, well, whether or not you're doing something slimy to make money.

Edit: Oh, and I just saw that you mentioned Capiche at the end. I just want to say that none of this comment is directed at them — I sincerely think that they're passionate, motivated founders who deeply care about doing good for the startup community.

[pc86]

I think it's possible to have a system where lesser-known products can pay to get reviewed, and have the review still be unbiased and truthful. The probably is that it's impossible to know where that stops and "pay 1x to get reviewed, pay 2x for a good review" begins, and I think the vast majority of people would be immediately distrustful of any system where I can pay for my brand new, single-customer startup to get reviewed by a giant like Gartner.