There was a paper at VLDB about Delta Lake: https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf - it describes why it was created, plus details of implementation.