[sumtechguy]

kafka has a 'schema registry' service. Typically they use avro/json to define the schema of each message. Think they added a couple of new types in recent versions. When you define your consumer/producer you also tell avro what the schema registry service URL is. Basically it helps you keep your messages in spec. It also has the ability for versioning so you can have different versions of messages in flight on the same topic. So if you add a new field it is fairly trivial for both sides to know what is going on. If you remove a field then it becomes more tricky.

Now what goes into that registry is typically in some way version controlled. That is for 'rebuilding' a clean environment if needed. Or part of a CI/CD system.

[kieranmaine]

Thanks for the reply. We're using a schema registry but we generally have our schemas spread across multiple repos depending on who owns the topic. I was wondering if they centralized all their schemas under a single repo. I'd like to do this, but I wanted to get some other opinions on the subject.

[sumtechguy]

I have seen several approaches.

1) the producer repo owns it. This has the benefit of only the producer makes those things and it is side by side. Downside if you have more than one producer repo with the same schema. When the producer code starts up it sets up the schema.

2) central repo. This is nice however tends to make it if you are using a micro service style more of pain to deploy as they all sorta need to move together. But works very nicely with mono repos. You can make it work with micro but it takes a bit more thinking. This turns your deploy into a two step process 'schema first' then code. That can work but again 'more thinking'.

3) consumer owned. The consumer basically says 'i will only grab these anything else I will consider error'. Works ok if you have 1 producer group and one consumer group and is effectively #1. But with several consumers it becomes a 'cut and paste job'. Or something may get out of sync, etc. When the consumer starts up it pushes the schema.

What I found best to get someone to decide is to figure out what is your version upgrade cycle. Is it one thing at a time? If it is 'everything goes every time'? Who 'owns the schemas?' So different styles will match your deployment process more than anything.

[kieranmaine]

Thanks for the feedback!

[linseed_213]

Evan can correct me if I'm wrong, but I believe it was one centralized service with all the schemas. You could view older versions and make new versions via the UI.

It was pretty user-friendly and managing schemas was straight forward. Haven't done anything similar since so no comparison point, but I thought it was fantastic and improved data quality a ton.

[evanrich]

This is correct yeah