Hacker News new | ask | show | jobs
by tene80i 493 days ago
Very cool! Can you share more about how you built it? I’ve seen a lot of skepticism about AI in scraping but this seems to work well.
1 comments

I'm glad you liked it! I asked ChatGPT to give me the URLs of exhibition directories of London's best museums and exhibitions. Then followed a content-based approach using BeautifulSoup and OpenAI GPT-4o structured outputs, targeting what kind of info I wanted to scrape: link, date, event title etc.

I was lucky that most museum websites were easy to scrape. Unfortunately it didn't work great for all of them, there I did some manual touches.