[DelTaco]

Oh wow I did something very similar, just dumber.

https://shopbysub.com

I have a raspberry pi gathering Reddit comments with an Amazon link in them every hour, then exposing them on Shop By Sub categorized by subreddit, sorted by upvotes.

Same with OP I put my Amazon referral link in there.

It's been decently well-received by Reddit, but I have a hard time getting recurring users. It seems like they just forget about it by the time they need a Reddit recommendation again.

Best of luck in the space OP!

[frontman1988]

Your site has a lot more subreddits than OPs. Wonder how you did that given there are literally millions(maybe even billions) of comments on popular subreddits. As for getting recurring users you can ask interested users to join a mailing list, you don't seem to have that on your site.

[DelTaco]

I've just been using a combination of the Reddit API and PushShift, warehousing data in such a way where it makes sense for Shop By Sub since Sept 2020.

Using the PushShift API, I started backwards in time and searched for a bunch of different regexes I thought were important, slowly creeping my way towards the present.

Using the Reddit API, I wrote a script to do a deep dive into specific subreddits, so I could index more popular ones first.

It's mostly just patience to get the amount of data I've got :) (and PushShift can't be overstated, it's very useful).

A mailing list isn't a bad idea. I'll have to figure out a way where the user can see a sign up box, but have it not be intrusive. I've been thinking of adding infinite scrolling to the most recents on the homepage, too, just for folks who are curious as to what weird things Reddit is talking about