[orliesaurus]

Very cool! I am curious about a few aspects:

How do you handle the scenario where errors may exhibit evolving characteristics over time, which might potentially impact the effectiveness of the current embedding model? Is there a mechanism for ongoing model adaptation to ensure sustained accuracy and relevance?

Lastly, could you envision a scenario where this technology is extended to not only classify, but also to predict potential errors based on historical and real-time data?

[vadman97]

Regarding model adaptation, we haven't yet explored a fine-tuned model, but it makes a lot of sense for a given class of errors. For a given code-base that is using highlight, the errors will typically be of a given language / infrastructure, so fine-tuning the model to those errors should be beneficial.

As for predicting potential errors, this will particularly make sense as an anomaly detection mechanism across metrics and logs. A class of 'important' errors based on the LLM's understand of the error, as well as historical comparison to normal error rates, is something we're exploring with an 'interesting errors' concept - stay tuned for more there!