[rafram]

Re type constraints: Not really. If one of the fields in my JSON is `name` but the model can’t read the name on the page, it will very happily make one up. Type constraints are good for making sure that your data is parseable, but they don’t do anything to fix the undetectable inaccuracy problem.

Fine-tuning does help, though.

[fzysingularity]

Yes, both false positives and false negatives like the one you mentioned happens when the schema is sometimes ill-defined. Making name optional via `name: str | None` actually turns out ensure that the model only fills it if it’s certain that field exists.

These are some of the nuances we had to work with during VLM fine-tuning with structured JSON.

[rafram]

You seem to be missing my point.