[volkk]

One thing that maybe is just wooshing over my head, but how does persistence fit into this, at least as best practices?

The way I understand this tool from the docs, is that a request is duplicated to go to the baseline service A as well as service A' -- so service A on `main` branch is actually serving the up to date code, while service A' has modifications that developers can quickly see either work or blow up. What happens if the change is a DB write change? Do both A and A' point to a primary prod DB and if A' changes something in that results in bugged data, wouldn't that screw up prod data? How do I go about accounting for that? Or am I just entirely misunderstanding the point of this tool?

EDIT: I think I just bumped into my own answer within the docs -- "Sandbox Resources." I see so far you have Mysql/Maria, SQS, and Rabbit plugins. What's next on the roadmap? Kafka/PgSQL soonish? :)

[anirudhrx]

I was just about to post this :)

Depending on the request context (typically a set of headers), a request will end up going to one of baseline service A or A’, but never both. Depending on the scenario being tested, you can choose to isolate additional stateful components. In the case of testing a change to the database itself (DDL, etc), you’re right in that the database must be isolated so that A’ does not talk to the same shared database as A. So, as you pointed out, typically this would involve using a resource plugin (https://docs.signadot.com/docs/using-resources-with-sandboxe...) to set up that new database shard (or table, or schema) that is isolated as part of the test sandbox environment, and then linking A’ to talk to that isolated instance instead.

Kafka is on our roadmap next! We're actually working right now on making custom resource plugins easy to write, because invariably with databases, there's a data seeding step that's pretty custom for users. Right now, for building a custom plugin, aside from writing the logic in the provision / deprovision scripts (for ex. https://github.com/signadot/plugins/tree/main/src/mariadb), there's a bit of docker image building and helm packaging needed that we're hoping to simplify.

[volkk]

> a request will end up going to one of baseline service A or A’, but never both

that's interesting, i totally didn't gleam that. so X percentage of users are getting a potentially bugged experience? is this kind of like a more robust feature flagging tool? i.e "rollout this thing to 1% of people for X time, and then roll it back to control"

and if so, if you have 200 engineers all testing all sorts of stuff on prod users/data, the chances of buggy experience for small amounts of people increases? am i understanding that correctly?

[anirudhrx]

Ah, the primary use-case here is for developers to test microservices as changes are made to them - so one would typically be running this setup in pre-production / staging only and no prod user data would be involved. I would think of it as a way to run integration / e2e tests for each commit or pull request. Running these tests in the high fidelity test environment constructed this way reduces the bottlenecks on staging environments, and bugs are discovered / addressed sooner.

[volkk]

AH! derp. thank you that makes total sense :) -- seemingly awesome tool, will certainly look into this. thanks for the quick responses