I got it working, with intermediate layers, too. All to find that I didn’t see that material a performance benefit after taking into account how long it takes to pull from and push to the cache.
You might want to try the actions cache "--cache-to=gha --cache-from=gha", but still it needs to pull that stuff down, just that locality is likely better here.
There's also an action out there "GitHub cache dance" that will stick your whole buildkit state dir into the gha cache.
There's also an action out there "GitHub cache dance" that will stick your whole buildkit state dir into the gha cache.