First, there are many gaps in pretty much every home. Junctions between drywall/plaster joints, windows and window frames, every electrical switch/outlet box, details in the attic and roof, the rim joists and sill plates, and yes, even any exposed masonry.
Secondly, exposed masonry does often warrant being air sealed: