Well, sure; if it didn't have walls, it wouldn't be a castle.
Some castles rely entirely on their own walls
Some castles rely entirely on their own walls