It doesn’t “restrict” to pypi but it wants to be rooted in pypi. That means if you install “tensorflow” you get it from there. The state of pypi is the state of pypi and I have some hope that this also will improve. See for instance the efforts that go into “wheel next”.
You can also import existing Python versions into uv, for example on my Mac it has imported the Homebrew versions.