[flexer2]

In principle I agree with the author, but some of the patterns these big companies have introduced are valuable for companies with significantly lower amounts of traffic. For instance, in the article they reference Kafka. In one of our products, we use Kinesis, which has similar semantics, for data that is no more than 25k records per day. However, we find it useful because it enables us to have multiple consumers that operate independently, plus using Kinesis Firehose to automatically archive those records off to S3. We just use a single shard, which is more than enough throughput for us. We don't have any plans to scale to hundreds of shards, but find what it provides to be very useful in separating what each process does, and makes the code much simpler. And if we ever did need to scale, it wouldn't be much work to do so.