This was 5+ years ago so I'm a little hazy on the details. A lot of it was just plugging into our infrastructure for config, service discovery, rate limiting, logging, etc.
- Logging the exact metrics we wanted to our metrics service. (We wrote a bunch of Lua code to do this.)
- Loading the set of backend services from Zookeeper. (We had a sidecar process that would periodically sync from Zookeeper to a config file, then call `nginx reload`.)
- Routing requests to different datacenters based on where the user's data was homed. (We had Go libraries to do this, and it would have been nice to be able to just call into that. I think we ended up putting this functionality in the sidecar.)
- Changing the way we streamed a response based on an HTTP header from the application. (We ended up writing a C module for this.)
- Changing the backend selection algorithm from "least connections" to "best of 3 random choices". Nginx added support for this in 2018, apparently.
There were ton of things we solved with Lua, which wasn't ideal. Since that's the only Lua in our codebase, everyone who touches it has to make a mental switch to remember all the weird edge cases (as you do with any language), plus 1-based array indexes, plus no static types.
Plus, any Lua or C you write has be contorted to fit Nginx's multi-stage request/response architecture. If you're writing a plugin that is meant to play nice with other plugins, then the contortions may be worth it. But for us, a proxy library would have given us the freedom to write code that was simpler and easier to understand.
- Logging the exact metrics we wanted to our metrics service. (We wrote a bunch of Lua code to do this.)
- Loading the set of backend services from Zookeeper. (We had a sidecar process that would periodically sync from Zookeeper to a config file, then call `nginx reload`.)
- Routing requests to different datacenters based on where the user's data was homed. (We had Go libraries to do this, and it would have been nice to be able to just call into that. I think we ended up putting this functionality in the sidecar.)
- Changing the way we streamed a response based on an HTTP header from the application. (We ended up writing a C module for this.)
- Changing the backend selection algorithm from "least connections" to "best of 3 random choices". Nginx added support for this in 2018, apparently.
There were ton of things we solved with Lua, which wasn't ideal. Since that's the only Lua in our codebase, everyone who touches it has to make a mental switch to remember all the weird edge cases (as you do with any language), plus 1-based array indexes, plus no static types.
Plus, any Lua or C you write has be contorted to fit Nginx's multi-stage request/response architecture. If you're writing a plugin that is meant to play nice with other plugins, then the contortions may be worth it. But for us, a proxy library would have given us the freedom to write code that was simpler and easier to understand.
The team moved to Envoy after I left: https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...