[techcode]

The other thing that's PITA with Kafka is fail/retry.

If you want to continue processing other/newer items/messages (and usually you do), you need to commit Kafka topic offset - leaving you to figure out what to do with failed item/message.

One simple thing is just re-inserting it again into the same topic (at the end). If it was temps transient error that could be enough

Instead of same topic, you can also insert it into another failedX Kafka topic (and have topic processed by cron like scheduled task).

And if you need things like progressive backing off before attempting reprocessing - you liekly want to push failed items into something else.

While it could be another tasks system/setup where you can specify how many reprocessing attempts to make, how much time to wait before next attempt ...etc. Often it's enough to have a simple DB/table.