[Aardappel]

Have you tried Teardown? Has incredibly good voxel physics. Definitely possible.

[HexDecOctBin]

Teardown calculates collision meshes and does physics on those. Not on voxels directly, and definitely on in voxel space.