It absolutely makes a difference. My neighbor on the same ISP as me pulling down a file from my FTP server does not have the same cost to my ISP as both of us streaming Netflix.
Yeah, and they charge you for your bandwidth to the internet, not your bandwidth to your neighbor. What travels along that bandwidth that you've already paid for is irrelevant. It costs the ISP the same amount whether they are sending you Netflix, or Youtube, or Hulu, or their own digital TV.