|
|
|
|
|
|
by JoshRosen
3817 days ago
|
|
|
The blog is referring to source- and binary-API compatibility guarantees that Spark provides. IIRC, in order for DataFrames and Datasets to share a common parent class certain common methods' return types would need to change in a binary-incompatible way that would break DataFrame code which was compiled against older Spark versions. Due to Spark's API compatibility and versioning policies, this type of compatibility break can only occur in major releases, so it can only be done in Spark 2.0.0 and not Spark 1.6. |
|
|