I see that you are scraping the results directly, curious if you’d tried out Google’s json search api? Any thoughts about it? I was about to need it or similar for a project.
Google's json search api gives you a consistent easy to parse api.
However it requires a Google API key, which limits free usage.
Client-side scraping scales effectively, as it distributes the reads onto all the clients. But it's also more brittle, as Google could change a small implementation of their search and break all the scraping functions.
I've tried it with the C# wrapper library, so using plain old methods and C# objects rather than touching the JSON directly.
It's not too bad, but not free as far as I'm aware and the result snippets they give you are quite limited in length. Also I found it a little tricky to get good results on some queries when I tried it out.
However it requires a Google API key, which limits free usage.
Client-side scraping scales effectively, as it distributes the reads onto all the clients. But it's also more brittle, as Google could change a small implementation of their search and break all the scraping functions.