Structural issues. Adds weight, in ways buildings aren't designed to take. Requires soil, and water in the soil, which is more weight. Water causes damage to building structure. Some plants will be impractical because they'll want to burrow into the structure. It's dangerous to collect the produce. There are insurance issues around stuff falling off onto people. It becomes difficult to clean or replace windows. Once plants are on it will be difficult to change damaged trellises. Change to fire risk profile of building. Security and insurance considerations from people climbing on the structures. Buildings not designed to supply water to the plants. Freehold issues from one person's plant growing into another's area. Allergies. Bees. Rats. Effect of plants growing into air-conditioning systems. Council complications over the change to the view profile of the building from sleek thing to tentacled thing.
Tons of upvotes for a comment that says hydroponic trellises require soil? What's going on here?
Hydroponics uses a fraction of the water that soil-based agriculture uses, and generally uses a very lightweight growing medium. (Vermiculite, hydroton balls, etc...)