[arynda]

How did you end up solving the rate limiting? Were the workers themselves responsible for coordination via a semaphore (e.g. there's already 25 of us, don't consume from this queue) or did you solve it on the scheduling side and only push jobs into their queues once you knew there was capacity?

[roflyear]

Scheduling side, we only schedule jobs if there have not been N of that type scheduled during that period. Works pretty good.

The jobs are still technically "queued" but they don't get consumed until something opens up (but workers can work on other tasks).

[arynda]

That's cool. I've been thinking through a similar design. Would this be an accurate description of your approach?

queues hold jobs, the scheduler/dispatcher handles load-balancing and rate-limiting, pushing jobs down to workers only when flow-control criteria is met?

  ┌────────┐                 ┌─────────┐
  │ queueA ├──┐           ┌──┤worker1  │
  └────────┘  │           │  └─────────┘
              │           │
  ┌────────┐  ├───────────┴┐ ┌─────────┐
  │ queueB ├──┤dispatcher  ├─┤worker2  │
  └────────┘  ├───────────┬┘ └─────────┘
              │           │
  ┌────────┐  │           │  ┌─────────┐
  │ queueC ├──┘           └──┤workerN  │
  └────────┘                 └─────────┘

[roflyear]

Yeah, pretty much! Technically I have one real queue but different "tags" to route things (so we can rate limit etc..). But basically the same thing.

My system supports as many dispatchers as you want, which is a good addition but makes the logic more complex as you have to be careful with locks so you don't schedule jobs many times, for example.

My system also implements retry logic, and a bunch of other stuff, but that isn't absolutely required either.

[calvinmorrison]

sorry, how did you draw this/

[arynda]

https://asciiflow.com/

[calvinmorrison]

thank you