I’m curious why you would do it this way vs publishing to SNS and having that fan out to multiple queues where each consumer can listen for the things it needs to work on (as mentioned in the original article.)
I found it just was not necessary for most cases. And the way I got there was working backwards from the "web scale" technologies like kafka, kinesis, and dynamodb.
I built a data ingestion system that handled an average of 300 messages/sec, peaking at 1,000, and writing to a single R3 RDS instance. You can do a lot by pushing simple scaling strategies to their limit. Everyone thinks they need to handle web scale, but really you just need to handle your scale.
I built a data ingestion system that handled an average of 300 messages/sec, peaking at 1,000, and writing to a single R3 RDS instance. You can do a lot by pushing simple scaling strategies to their limit. Everyone thinks they need to handle web scale, but really you just need to handle your scale.