[testplzignore]

The most telling fact is that it "took milliseconds to complete running the tests for all 140+ of our destinations". I've never worked on a single service whose tests ran that fast, given that the time spent by the overhead of the test framework and any other one-time initialization can take a few seconds just itself. It's great to have tests that run fast, but that's a bit ridiculous.

Some rules of thumb I just came up with:

Number of repos should not exceed number of developers.

Number of tests divided by number of developers should be at least 100.

Number of lines of code divided by number of repos should be at least 5000.

Your tests should not run faster than the time it takes to read this sentence.

A single person should not be able to memorize the entire contents of a single repo, unless that person is Rain Man.

[mpweiher]

> never worked on a single service whose tests ran that fast

I'd say you've never had good tests.

I have a test-suite for a bunch of my frameworks that dates to the mid 90s, with tests added regularly with new functionality.

It currently takes 4 seconds total for 6 separate frameworks and 1000 individual tests. Which is actually a bit slower than it should be, it used to take around 1-2 seconds, so might have to dig a little to see what's up.

With tests this fast, they become a fixed part of the build-process, so every build runs the tests, and a test failure is essentially treated the same as a compiler error: the project fails to build.

The difference goes beyond quantitative to qualitative, and hard to communicate. Testing becomes much less of a distinct activity but simple an inextricable part of writing code.

So I would posit:

Your tests should not run slower than the time it takes to read this sentence.

[mekazu]

Unit tests that don’t read or write to disk and don’t try thousands of repetitions of things should be bleeding fast, but the most useful integration tests that actually help find faults (usually with your assumptions about the associated APIs) often need interaction with your disk or database or external service and tend to take a bit more than a few seconds. I find you need both.

[inetknght]

I have tests which verify DNA analysis. The test data vectors are large -- a few hundred MB here, a couple GB there. The hundreds of tests that use these test vectors still run in a few seconds.

If you're using a tape drive or SD cards, sure. But even a 10 year old 5400RPM on an IDE connection should be able to satisfy your tests' requirements in a few seconds or less.

I suspect your tests are just as monolithic as you think microservices shouldn't be. Break them down into smaller pieces. If it's hard to do that, then redesign your software to be more easily testable. Learn when and how to provide static data with abstractions that don't let your software know that the data is static. Or, if you're too busy, then hire a dedicated test engineer. No, not the manual testing kind of engineer. The kind of engineer who actually writes tests all day, has written thousands (or hundreds of thousands) of individual tests during their career. And listen to them about any sort of design decisions.

[dvogel]

Sounds like you have tests that need to read (probably cached) data files while the parent poster has tests that need to write to disks (probably in a database transaction). Those are different enough that run times won't ever be comparable.

[inetknght]

I have tests that need to read. I have tests that need to write. All data written must also be read and verified. You're right, the data is probably cached.

If you need to access a database in your tests you're probably doing it wrong. Build a mock-up of your database accessor API to provide static data, or build a local database dedicated for testing.

[mpweiher]

Sure. I'd venture to say that integration tests should be fewer than unit tests, see hexagonal etc. Hopefully those external interfaces are also more stable, so they don't need to be run as often.

I tend to use my integration tests also as characterization tests that verify the simulator/test-double I use for any external systems within my unit tests.

See also: the testing pyramid[1] and "integrated tests are a scam"[2], which is a tad click-bait, but actually quite good.

[1] https://martinfowler.com/bliki/TestPyramid.html

[2] https://www.youtube.com/watch?v=VDfX44fZoMc

[sieabahlpark]

Not all domains have libraries which can run tests that fast?

[mpweiher]

Sure. I'd suggest that the domains that limit testing speed are exceedingly rare. Much, much, much more common are technical issues.

Either way, the idea of considering slow tests a feature was novel to me.

[inetknght]

> I've never worked on a single service whose tests ran that fast, given that the time spent by the overhead of the test framework and any other one-time initialization can take a few seconds just itself. It's great to have tests that run fast, but that's a bit ridiculous.

It's not ridiculous. It's good.

I work on an analysis pipeline with thousands of individual tests across a half dozen software programs. Running all of the tests takes just a few seconds. They run in under a second if I run tests in parallel.

If your tests don't run that fast then I suggest you start making them that fast.

I'd be willing to bet that if you learned (or hired someone with the knowledge of) how to optimize your code, you could get some astounding performance increases in your product.