I can delete songs from my iOS 7 phone by swiping. I'm not sure which settings factor into this - it's being synced with iTunes via USB, and it's logged into iCloud but I doubt that makes a difference.
Strange, I can't get the swipe behavior in the Music app. I've yet to plug the phone into iTunes since the iOS 7 update, but whenever I swipe across an album or song, it acts as if I tapped it.