[diogofranco]

csv is notoriously bad with spark - in fact you couldn't write to csv for the longest time without external libraries like the databricks one, so I don't contest your last point.

SQlite is usually a single file representing a database though, I don't know how it would work with partitioning and stuff, and then how to handle the schema evolving across sqlite files.

[drej]

Also, the now native Spaek CSV export doesn’t follow RFC 4180 with default settings, it escapes double quotes with a backlash. So odd.