Hacker News new | ask | show | jobs
by candiddevmike 961 days ago
Why have consumers and producers vs doing it all in one goroutine, utilizing some kind of connection pool?
2 comments

Because then you are consuming, or producing, you can’t do both at the same time. You are either reading from a stream of data, or you are writing it. Using goroutines to separate these allows you to do both at the same time, as soon as data is available on the channel or you receive the signal to stop.
To get higher throughput we would need one goroutine to pull from the replication slot while the other is pushing to the target. The idea is to keep the Postgres connection useful and reading the slot while also pushing to the target asynchronously.