What do you mean here by memory layout? For instance, the order of fields in a rust struct can (theoretically) change by recompiling. It's not defined by the order of fields in the definition.
On a language level, high level APIs will necessarily contain details to things like (mut) reference, Box whatever. Which is not a problem at all, given the problem domain, but in my opinion it is not possible to make a both low and high level language at the same time (and it is not really needed either)