The reference implementation is not the same as knowing which code runs on "your" hardware and having all keys from "your" hardware. If you are protected by a steel door but you don't have the keys, you are not safe, you are imprisoned.
This seems like a ridiculous argument. There is a fully source open source secure boot implementation, that if you have concerns about it you can fully audit, just as you can fully audit your current setup if you wanted to - but almost certainly have not.