As stated in the manifesto "while there is value in the items on
the right, we value the items on the left more."
That doesn't mean there isn't room or need for contracts. A contract can and should exist with clear expectations of the customer's involvement and availability with customer response and turn around times and what the consequences are if the customer doesn't fulfill their end of the "bargain".
If you go to the customer and the customer doesn't answer your questions, that is obviously a huge problem. Then that customer is not ready to work agile. Maybe the situation can be salvaged, maybe not. If not, then apparently it is not the customer you would want as an agile team.
The manifesto intended that your customer can (and often will) be a proxy. You need a PM who gets out of the office, empathizes and understands clients, talks to customers. A big part of what's wrong with "agile" today is BS product managers.