I'm clearly not the target audience for this, so excuse me if this is a dumb question: what is this tool used for? Who would usually use it and for what purpose?
Are you continuously monitoring consistency proofs? Or in other words, would someone (you or someone else) actually notice if a log changed its contents retroactively?
I tried to do something like this one time and had a problem just finding the logs. All information on the internet points to the fact that certain logs exist, but not how to access them. Are they not public access? Do you have a B2B relationship with the companies like Cloudflare that run logs?
They're required to be public services. https://crt.sh/monitored-logs is the list of logs monitored by crt.sh (a public log monitor operated by Sectigo, a commercial CA) if that helps. Each of the major browsers also publishes which logs they trust and provides information about e.g. distrust of logs. Is the problem that you couldn't figure out how to use a log? It doesn't just have a web site where you can type in searches you need to be able to use their web API as defined in the protocol documentation.
This is really cool! It discovered even subdomains that lived for a few days on my site. If it’s not a secret, how do you discover those? Is it by listening to DNS record changes?
Merklemap is running PostgreSQL as the primary database, currently scaling at ~18TB on NVMe storage, and around 30TB of actual certificates that are stored on s3.
The backend is implemented in Rust (handling web services, search functionality, and data ingestion pipelines).