Log4j2 uses some JVM features to resolve some addresses, and these features can end up blindly loading external classes in the affected JVMs.
Note that non-affected JVMs are still vulnerable to other issues triggered by that resolution process, just not as bad as loading untrusted remote objects. So you should upgrade log4j2 even if you use a non-vulnerable JVM.
Newer Java versions disable deserialization of remote classes via LDAP. You're still vulnerable to deserialization of existing classes, but to exploit that there have to be exploitable classes on the classpath already.
Note that non-affected JVMs are still vulnerable to other issues triggered by that resolution process, just not as bad as loading untrusted remote objects. So you should upgrade log4j2 even if you use a non-vulnerable JVM.