| Show HN: StreamGate – A lock-free observability proxy in Go Hey HN, I built StreamGate to solve a specific problem: our observability bills (Datadog/Splunk) were scaling linearly with traffic, mostly due to low-value "noise" logs (DEBUG, health checks, etc.) that we rarely queried. Existing solutions were either "all-or-nothing" agents or heavy Java/Enterprise pipelines. I wanted something lightweight that could sit at the network edge and act as a smart valve. The Architecture: It uses a Split-Plane design: Control Plane (Python): Handles config validation and API requests (FastAPI). Data Plane (Go): Handles the hot path. Technical Implementation Details: Instead of standard Go Channels (which I found created too much GC pressure at high throughput), I implemented a fixed-size Ring Buffer using sync/atomic primitives. Lock-Free: The hot path uses atomic pointers to swap configuration rules, so we can "hot reload" rules without stopping the world or acquiring a Mutex. Fail-Open: It implements a circuit breaker; if the ring buffer fills up (>80%), it degrades to pass-through mode rather than dropping logs or blocking the app. Performance: On my local machine (M2), it handles ~200k events/sec with minimal allocation. I’d love critical feedback on the ring buffer implementation or the split-plane approach. |