[chriswarbo]

> And the answer is pretty simple: pin the specific test repo version! Use lockfiles, or git submodules, or put "cd tests && git checkout 3e524575cc61" in your CI config file _and keep it in the same repo as source code_ (that part is very important!).

Indeed. Where I work we have a bunch of repos, but they always reference each other via pinned commits. We happen to use Nix, with its built in 'fetchGit' function; it's also easy to override any of these dependencies with a different revision. For example:

  { helpers ? import (fetchGit {
      url = "git://url-of-helpers.git";
      ref = "master";
      rev = "11111";
    })
  , some-library ? import (fetchGit {
      url = "git://url-of-some-library.git";
      ref = "master";
      rev = "22222"
    }) {}
  }:
  helpers.build-a-service {
    name = "my-service";
    src  = ./src;
    deps = { inherit some-library; };
  }

This is a function taking two arguments ('helpers' and 'some-library'), with default arguments that fetch particular git commits. This gives us the option of calling the function with different values, to e.g. build against different commits.

We run our CI on GitHub Actions, which allows some jobs to be marked as 'required' for PRs (using branch protection rules). The normal build/test jobs use the default arguments, and are marked as required: everything is pinned, so there should be no unexpected breakages.

Some of our libraries also define extra CI jobs, which are not marked as required. Those fetch the latest revision of various downstream projects which are known to use that library, and override the relevant argument with themselves. For example, the 'some-library' repo might have a test like this:

  import (fetchGit {
    url = "git://url-of-some-library.git";
    ref = "master";
    # No 'rev' given, so it will fetch 'HEAD'
  }) {
    # Build with this checkout of some-library, instead of the pinned version
    some-library = import ./. {};
  }

This lets us know if our PR would break downstream projects, if they were to subsequently update their pinned dependencies (either because we've broken the library, or the downstream project is buggy). It's useful for spotting problems early, regardless of whether the root cause is upstream or downstream.