yea definitely not cheap, to break it down further:
I first ran each post title through gpt-3 to classify as either likely or unlikely to be insightful to founders. Then, if it is likely to be insightful I pass the entire post content including up to 100 comments into gpt 4.
This comes down to around 13-14 million tokens used for the 15k posts, which for the latest gpt-4 model is ~$150.
Im sure this could be done cheaper with something like Llama 2 but tbh I am fine with this for now since I was aiming for speed with the initial list to gauge interest :)
I look at your screenshot list and those are huge problems with huge companies trying to solve them. This seems more like a big opportunities startup list than something for side projects. That may not be representative of what is in the database, but those are huge pain points, not niche problems.
yea great point, I chose those in the screenshot since they caught my eye when scrolling through. But in general yes I think i need to work on the database more to only contain the super niche opportunities
I didn't realize the first round that it was actually just a nice and neat list of scraped information, which I am quite happy to spend time going through. Just purchased a membership.
thanks for the insight! - yes absolutely this is not a list of validated ideas by any means, more so a feed of inspiration and hints towards market interest (if you find 5-6 reddit posts complaining about a similar thing it has higher likely hood of validation then random idea you think of in the shower). I'm working on a free plan which gives 10 a day to scroll through so currently looking at ways of increasing the value here beyond just the static list
Im an indiehacker who loves to solve for niche pain points
I realized I spend a lot of time just scrolling through subreddits looking for user complaints, insights, etc
I decided to have ChatGPT do the heavy lifting for me, and consolidate a more curated list of posts with insights for me to read
Stats:
Posts Processed: ~15k Insights Found: ~1K API Costs: ~$150
I hope a few of you may find this useful, would appreciate any feedback
~Justin