But secrets are a kind of configuration, right? I agree they should be kept separate from non-secret config; ideally non-secret configuration can be committed to your codebase somewhere depending on the environment it's running in.
pydantic-settings is not limited to loading from config files. It is easy to populate a settings model from environment variables, for example. With a little bit of glue code it could load from the secretspec sdk.