To add to this, often cafes enforce a minimum transaction size, such as $10, to use card. This means usually you only use card it you are grabbing lunch as well.
I've read before[0] that it is actually a violation of a stores terms with Visa to enforce minimum transaction sizes. Those stores could potentially be reported to Visa and have sanctions placed on them (like an increased transaction rate). Interestingly, a merchant is allowed to offer a discount to people who pay by cash but they can't charge more for people paying by credit card.
Yeah it is annoying really. Especially for anywhere where it is common to buy things under $10. I think in the end most businesses would be better off just taking the fee hit as they would likely see increased business.
[0] - http://www.msnbc.msn.com/id/11697094/ns/business-consumer_ne...