[stevem1978]

I just signed up because of this thread. this is something I did 12 months ago. I only did northwest IDOX including Wirral and Liverpool. It was a long slog so i salute you.

I am not experienced at this at all but I managed to get around 107k from Wirral. the parsing afterwards was a bit crap. I could probably dig out the code from somewhere. the approach i took was quite long winded:

created a txt file with ID numbers 100000 to 200000 (or similar) then i think i used playwriter to pick a random number and scrape it then write the details to a CSV. then in another script i checked the CSV to see if the decision date was present and if it wasn't then repopulate the txt file. a massive pain and it of course changes every week!

I can think of a good use case and people who would be interested in this data though. I would love to see your work, especially for IDOX. don't give up, as others have said change your business model. members of the public are not your audience.

I dont think. look at selling the whole database rather than one line item. you will probably copies of the PDF decision notices etc. Oh and people who say that you should do a FOI.... ha! you would need to FOI every single house and street in the borough. the work that would create for the council would be expedential. The council charge businesses to access this free data. Councils should make the data available as a free download hosted. then all this would stop.