Short answer: no. In theory, it would be possible to increase the speed, but that would include a rewrite of these two font packages. Fontloading is not necessarily slow in Lua(TeX), but the mentioned luaotfload for example deals with all kinds of complex scripts which is not necessary in most cases.