[sowbug]

Thanks for the analysis. Something still sticks in my craw.

You can read a letter. You can play an audio recording. You can look at a photograph. You can print a word-processing document. Those feel like static records. Each tells a single story, no matter how many times it's read.

An ML model, on the other hand, is just a long sequence of floating-point numbers. There is no meaningful model "viewer" that spits out a text file or JPEG. The only thing a non-engineer human can do with an ML model is interact with it when it's running. Thus, any output is the product of both the human and the model. It doesn't feel like a record. It feels more like a performance -- and a performance by the interrogator, at that.

If an ML-model "record" is a special kind of record that needs manipulation to produce human-readable output, then it feels like we're back in the 5th amendment department. No, chatting with a chatbot is not torture. But it produces a kind of evidence that is an unwelcome collaboration between the interrogator and the record (unwelcome by the owner of the record). It's one thing to let a jury look at a screen full of numbers. It's another thing to say that an expert witness used those numbers in a chat session that printed "Yes, it was my owner, in the living room with the candlestick."

If you lock up a person and ask them the same question over and over again, eventually you'll get the answer you want. The same will probably turn out to be true for that person's chatbot. Should society allow that second case?