[mholt]

No, Caddy never did that. Caddy never stopped serving a site that it started serving. Only by shutting down the server (or sending the appropriate signal) would it stop serving the site.