[chrsig]

I've been having to put together a test suite at $work similar to what the product tries to offer. If there were a sufficiently advanced product to buy, I'd petition for it to be adopted at my org.

> It is random sampling. I feel, 1M or 10M randomly sampled requests should cover all cases.

1. I suggest providing alternate approaches to sampling: The input itself may have bias towards a single use case. If 70% of the input exercises the same code path, there's no benefit to having a uniform sample. Ideally it would be stratified amongst customers, or perhaps on other dimensions to allow for covering the most surface area.

2. Requests don't happen in a vacuum. They likely have data dependencies on prior requests. I recommend some way of sampling sessions rather than individual requests. Replaying the 3rd request in a series of 6 is likely just going to be exercising failure paths.

3. Behaviors may vary between requests with respect to time. If requests were sampled over a number of days but replayed within a short time period, there are behaviors that could differ from what actually occurs in production.

I didn't see any explanation on how results are determined. I think it's important to surface those types of details on the website. I'm not going to watch the video on it in hopes of learning.

[vedant_ag]

> 2. Requests don't happen in a vacuum. They likely have data dependencies on prior requests. I recommend some way of sampling sessions rather than individual requests. Replaying the 3rd request in a series of 6 is likely just going to be exercising failure paths.

This is so true. I ran into this problem when I was trying to implement a kind of sampling that limited total RPS. Thanks for reminding. I am definitely looking into more sophisticated methods of sampling.

> I didn't see any explanation on how results are determined. I think it's important to surface those types of details on the website. I'm not going to watch the video on it in hopes of learning.

Good feedback! :)