[specialist]

True.

My ETL work in healthcare finally pushed me to treat the problem as screen scrapping. Bypassing all the attempts at formality, eg XSD.

The next step, which I prototyped but was put into purgatory by being acquired, was to simply capture all the data and use text retrieval tools, eg Lucene. Versus ingesting the data, normalizing it (to some schema) and populating database(s). Basically, postponing the translation/transformation of client data until viewing. Because, as you hinted, everyone does stuff differently, and clients generally have no idea what their data looks like until we showed them.

The proper solution would be to have direct access to source data, versus data feeds, but that ain't gonna happen until we have single payer, because the current incentive structure strongly discourages such simplicity.