Hacker News new | ask | show | jobs
by EdwardDiego 3109 days ago
> Amusingly, Kafka Connect wanted to use a queue with the same name as my table (MySQL or Hive / Hadoop, I don't recall which end)

It'll be the MySQL end if it's a Connect source as opposed to sink.

Two options - in your Connect config, you can specify a topic prefix, or if you use a custom query, the topic prefix will be used as the entire topic name.

> It was around about then that I figured out that Confluent was a bunch of kids playing at building stuff.

Kafka Connect saved me writing a load of boilerplate to monitor a PG database to propagate model updates in a medium suitable for streaming jobs - Kafka Connect + Kafka Streaming's Global KTables is a nice fit, even if the Connect JDBC end is somewhat beta at this point (KTables rely on Kafka message key for identity, the JDBC source doesn't populate it by default, so you have to use Single Message Transforms (SMTs) to achieve it)

I'd say beta, not kids.