[Fission]

Actually, I totally agree with you. This is something that we thought hard about, because our initial three guides are dominated by YC companies.

I think the main explanation is that YC companies, to a much greater extent than other companies, tend to focus on early-stage startups. As our guides are currently focused on early-stage startups, we naturally see more YC companies represented in the guides.

If anything, I'd argue that we're so aware of this that we need to consciously avoid overcorrecting in the other direction.

[ijpsud]

Have you thought about collecting reviews from "real" users of the products to supplement your analysis?

There's potentially a misaligned-incentives problem with review sites which themselves do the reviews. The latest episode of the FYI podcast[0] goes through this problem with the founder of Capiche[1] (looks like a competitor of yours) and the founder was explaining how firms like Gartner basically have two sets of customers: The software buys and the software sellers. The software sellers can actually pay Gartner to get in front of an analyst and sell them on their product.

I don't know to what extent this makes Gartner's recommendations less useful, but it seems like it would make it hard for smaller companies to get on the radar. I think the fundamental problem is that the analysts are not actual users of the software.

So I like Capiche's direction, but I dislike the fact that they're a "closed" invite-only website.

[0] https://twitter.com/ARKInvest/status/1261421627717218305

[1] https://capiche.com

[awwstn]

Thanks for mentioning Capiche! I think there's so much opportunity in this space so I'm stoked to see anyone else joining in.

At the very start, I wanted Capiche to be essentially Wirecutter for SaaS. We ended up going in a different direction, but I'm super impressed with Satchel's execution here, and as someone who spends a crazy amount of time on this stuff, I learned a bunch of new things reading through the site today.

I really don't want to hijack @fission's launch discussion, but thought I'd just reply to your comment on the invite-only thing: our site will be wide open soon-ish. Just taking our time building a community around high quality discussion (most of which has little to do with comparisons/reviews), but eventually will open it all up. Think of it like Stack Overflow's private beta...just a step in the process.

[Fission]

Hey Austin, I appreciate the thoughtfulness and the kind words.

I think even though we're working in different directions, we share a lot of the same values and see similar issues with the status quo. When (I hope not if) quarantine's over, let's grab a coffee :) — my email's in my profile

[1cvmask]

Any chance we can join now?

[Fission]

I think there are a couple of great points here, and I'll try to break things down and address each point independently.

1. re: "real" user reviews — right now, we do do this, just in a different way vs. what I think you're thinking of. For all of our guides so far, we've conducted a lot of interviews with founders who've used each product. However, we ultimately synthesize this data, pair it up with our own findings from our testing, and draw our own conclusions.

We think this is close to the optimal way of doing things. We found that sites which rely 100% on user reviews usually hit a ceiling in terms of usefulness, as the reviews are typically on the shallower side, which compounds b/c high-level reviews typically end up talking about the same things, so they're often redundant as well. By getting founder feedback, we're able to get a lot of datapoints to guide our findings and to surface less common issues, but by testing and ultimately writing these guides ourselves, we're able to both spend the time to elaborate on important aspects in depth (e.g. caveats, background info, etc.), while also being able to present info from a bigger picture perspective (e.g. this is how X, Y, and Z compare, etc.). The main downside is that it requires a lot of additional work.

2. re: Gartner + incentives — I believe that you identified the problem correctly, but misattributed the underlying cause. Gartner's incentives are misaligned b/c vendors who pay have an edge up on those who don't, for reasons you correctly identified. Importantly, this is not isolated to Gartner — in fact, the incentives are just as misaligned for a lot of the 100% UGC review sites. For those sites, if a vendor pays, they're allowed to cherrypick their own users to invite for an "organic review." If the vendor doesn't pay, then their rating will be relatively lower b/c it becomes much harder for them to get their best users to write a review on the site.

So in general, the problem isn't caused by whether you're a UGC review site or write guides yourself — it's caused by, well, whether or not you're doing something slimy to make money.

Edit: Oh, and I just saw that you mentioned Capiche at the end. I just want to say that none of this comment is directed at them — I sincerely think that they're passionate, motivated founders who deeply care about doing good for the startup community.

[pc86]

I think it's possible to have a system where lesser-known products can pay to get reviewed, and have the review still be unbiased and truthful. The probably is that it's impossible to know where that stops and "pay 1x to get reviewed, pay 2x for a good review" begins, and I think the vast majority of people would be immediately distrustful of any system where I can pay for my brand new, single-customer startup to get reviewed by a giant like Gartner.

[shostack]

I'm not sure I follow. Presumably, more mature SaaS products have more widely available reviews, documentation, tutorials, complaints, etc. to help companies make more informed decisions.

They also seem like they might have more proven business models, which is important when you're relying on it for a core function of your business. The last thing you want is some fresh startup that craters and forces you to scramble to find and implement a replacement when they fail to close their next funding round and need to shutter things.

[Fission]

Would you mind elaborating on what you don't follow on? I generally agree with everything you've said, so normally it'd be A-OK, but I think there's a question here somewhere (though I'm not exactly sure what) and I want to make sure I can address it.

[OoTheNigerian]

It's more of a comment on your previous response than a question.

1. Older companies tend to have wider footprint of real world use in various contexts thereby having a wider "body of work" for objective analysis and review.

2. There is switching cost when you get the 'thank you from the incredible journey" email or blogpost. So it is important to have companies that are self-sufficient and aren't at the risk of being acquired and shutdown which new companies are much more prone to.

PS: It is acknowledged that new companies most likely have fresher takes on old problems but you should also cater for their negatives enumerated above.

[shostack]

The other person who responded nailed it.

There's definitely value and opportunity that can be found in using the newest tech from the freshest companies. You can often find great value from their pricing, better UI, improved results, etc.

So my comment was in no way meant to bash using a startup. It was meant more to point out that there's a double-edged sword here by focusing on startups as there are real risks and trade-offs.