[13pixels]

The JS rendering point is critical. Even though bots like GPTBot technically have headless capabilities, they often fall back to text-only extraction for non-priority pages to save compute. We see a lot of "invisible" content in e-com especially because of this.

One other signal to check: internal linking structure. AI crawlers seem to respect semantic clusters more than traditional pagerank flow. If your "about" page isn't semantically linked to your "product" page in a way the LLM understands as a relationship, it often hallucinates the connection.

[apswin]

Thanks for the detailed feedback. Those are the next items on my list now. Will add headless browser research capabilities to go around java script issues. Will also add semantic clustering check.

Seems like you are quite well versed with the space. Would you be open to sharing some interesting resources or getting on call with me to share if you have struggled with this problem and what your workflow looks like?