[csuper]

I think you should stick with the framework, at least until your requirements exceed the frameworks capabilities.

There is always a simpler solution that you can one off. But is it worth your time to reinvent the wheel?

Also, I'm curious - were you able to do the same thing in Python that you did in Scrapy because you tried Scrapy first?

[zeynel1]

Yes, I was thinking the same thing. Scrapy has good documentation but still it didn't work for me. With Python it was easier and I understood what each piece of code did.