| I built FastWorker after getting tired of deploying Celery + Redis for simple background tasks in FastAPI apps. Every time I needed to offload work from API requests, I had to manage 4-6 separate services. For small projects, this felt like overkill. FastWorker is a brokerless task queue requiring only Python processes. No Redis, no RabbitMQ – just 2-3 Python services instead of 4-6+. --- Quick example: # tasks.py from fastworker import task @task def send_email(to: str, subject: str): return {"sent": True}
# FastAPI appfrom fastworker import Client client = Client() @app.post("/send/")
async def send_notification(email: str): task_id = await client.delay("send_email", email, "Welcome!")
return {"task_id": task_id}
Start workers:fastworker control-plane --task-modules tasks fastworker subworker --task-modules tasks # optional --- Architecture: Uses NNG messaging for direct peer-to-peer communication. Control plane coordinates task distribution via priority heap and tracks worker load. Workers auto-discover via discovery socket. Results cached in-memory with LRU/TTL. Designed for: Moderate-scale Python apps (1K-10K tasks/min) doing background processing – image resizing, report generation, emails, webhooks. Great for FastAPI/Flask/Django. NOT for: Extreme scale (100K+ tasks/min), multi-language stacks, or systems requiring persistent task storage. For those, use Celery/RabbitMQ/Kafka. Try it: pip install fastworker Repo: https://github.com/neul-labs/fastworker FastAPI integration docs: https://github.com/neul-labs/fastworker/blob/main/docs/fasta... Would love feedback on whether this fills a useful niche or if the limitations make it too narrow. |