[jannes]

Did you ever have issues with local DNS caching after switching location?

Did you ever have to run the following command to reset Apple's local DNS after switching?

    sudo dscacheutil -flushcache
    sudo killall -HUP mDNSResponder

As far as I know, Pi-Hole sets a relatively short TTL on its responses, but I think should still cause a non-zero delay when you disable it, shouldn't it?