[tikazyq]

There are a couple of crawler management projects: scrapydweb, spiderkeeper, gerapy, crawlab. The first three are based on scrapyd.