[fjni]

> I already knew from previous experience that for some reason, AT&T traffic to fast.com is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress

This I think is because they throttle ip addresses for known video streaming sites. That is one of (the only reliable) ways an ISP can get the streaming provider to drop the stream to a lower quality one by default. Since fast.com is a Netflix ip, and the isp can’t distinguish whether it’s video that is being transferred or a file to measure throughout, the speed test gets caught up in it. Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason.

[avianlyric]

> Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason.

Yup. This is pretty much the reason Netflix created fast.com. They wanted a speed test service that couldn’t be gamed by ISPs. Many ISPs will prioritise traffic to know speed test services (like Ookla’s speedtest.net), making their services appear faster than they’re under more normal usage.

By placing fast.com on Netflix IPs, ISPs either have to prioritise all Netflix traffic (which they’re very unlikely to do), or accept that fast.com is going to provide a more realistic measurement of their performance.

[neurostimulant]

Now that my ISP bundles Netflix subscription into their internet plans, access to netflix and fast.com now practically saturate the fiber link, while before it was outright blocked. Hooray for no net neutrality I guess.

Another fun part: when netflix IPs was blocked by this ISP, it's pretty much impossible to use netflix because the only way to get around the block was to use VPN, but netflix itself blocks VPN access.

[patmcc]

Your ISP entirely blocked netflix? That's incredible shitty.

[baby]

Heh, just revert back to pirating. That’s what I do whenever things don’t work.

[BLKNSLVR]

From the above thread it looks as if piracy might actually be a more bandwidth economical option than streaming services. Spread the load of the bandwidth around a bit more. Download any time other than peak streaming hours to then watch, self-contained, during peak streaming hours.

[mr_toad]

Plus you can rewatch without downloading again. And use sneakernets for sharing with your friends.

[dhosek]

Early in the pandemic, I spent a while without wired internet using a wireless hotspot from the library which would not connect to Netflix but any other streaming video service was fine. I forget who the wireless vendor behind the hotspot was—I think it might have been Verizon.

[kerkeslager]

So there's a bit of a weird fact about public libraries, which is that it probably wasn't on the normal internet (Internet1) but was in fact on Internet2 (see [1]).

Internet2 is used by hospitals, among other things, and as such has higher robustness requirements than Internet1. The pandemic created much higher demand for bandwidth from hospitals, forcing Internet2 providers to scramble to keep up in areas. As such, blocking high-bandwidth sites which are clearly lower priority than medical traffic might have actually been a reasonable move.

[1] https://en.wikipedia.org/wiki/Internet2

[bluedino]

I've worked at two hospitals. Both of them "had" Internet2. I was excited, as I had not heard about it in ~20 years.

Neither of them actually ran any traffic over it.

[riceart]

Not quite sure I understand - what do hospitals need a ton of bandwidth for? Why would those bandwidth requirements rise significantly during the pandemic. Sure there were a bunch of people on vents in the MICU, but pretty much every elective procedural service plummeted.

[Spooky23]

Spectrum used to dump all Netflix traffic to a device on their regional network. If you blocked that IP range, the service would perform dramatically better.

[dspillett]

> Now that my ISP bundles Netflix subscription into their internet plans, access to netflix and fast.com now practically saturate the fiber link

They may have an ISP-local netflix cache, from Netflix themselves not some home-grown hack, so they can achieve that with some reliability without it costing as much as it otherwise would for bandwidth peering.

[tiedieconderoga]

Netflix distributes "red boxes" to ISPs which are exactly that: edge content caches.

(PDF): https://openconnect.netflix.com/Open-Connect-Briefing-Paper....

[postdb]

BTW, AT&T has terrible records for dropping customer's traffic and also email without warning or notices to their customers. Why such terrible company is around amazing how corporation don't really survive by merit in our current world.

[solarpunk]

I'm curious where you live and what provider does this weird Netflix reselling practice.

[neurostimulant]

Streaming services do this all the time in Asia. Instead of paying with credit card, you pay via your carrier/ISP, sometimes at discounted pricing. I paid Netflix, Disney+, Amazon Prime and Zoom this way.

[callahad]

Common in America (e.g. https://www.t-mobile.com/tv-streaming/netflix-on-us) and the UK (e.g. https://news.virginmediao2.co.uk/virgin-media-o2-revamps-top...)

[happymellon]

> Hooray for no net neutrality I guess

Surely it would essentially saturate your fibre if there was net neutrality, unless you don't pay for full fibre speeds?

[tpxl]

Parent was being facetious.

[bee_rider]

What is the internet anyway? Are there truth in advertising issues around selling internet access with excessive filtering?

[pratnala]

Netflix blocks datacenter VPNs but you can use residential VPNs (which tbf are more expensive) to circumvent this.

[noobcoder]

Mine did the same and charged for bundles of HBO Max, Netflix, Disney+ Until recently they reverted it after bunch of complains and tonnes of people changing their ISP

[postdb]

hmmm, IMHO, traffic for video and everything else ought to separated into 2 classes. Video obvious takes much much more bandwidth than other stuff. It would be good to allow none video, smaller assets that are needed to to make a website function to have priority over video files.

[anonymousnotme]

It definitely looks like testing sites are prioritized. The fastest download speed that I have gotten is maybe 7 MB (bytes) per second; generally it is 2-5 MB per second. The speed test sites generally get 100 Mb per second dowload. In general the best I seem to get is about half the speed of the speed test sites. To me the real speed of the ISP is how fast one can download something one wants, not the result of a test. I would prefer to see results for downloads/uploads from youtube and various CDN networks and popular sites. I would also like to see ISP have a URL that is inside their network to test upload and download so that one can at least isolate what part of the connection might be lagging. Actually, I just used devtools to snag a 25MB file from fast.com. Curl/wget gives a speed of about 3 or 4 MB per second. That does not really seem to match up with fast.com download speed of 70Mb/second. 70/8 is 8.75, which is about double. Is fast.com accurate? Is my math wrong?

[jackson1442]

I personally like https://speed.cloudflare.com since it just looks like you're doing typical CloudFlare traffic. The results viewer is also quite nice.

[binkHN]

Very cool! I never knew about this! I really appreciate the latency during upload test. I've bookmarked this and will stop using speedtest.net. Thank you!

[is_true]

My ISP (Telecom) gives me a speed of 250Mbps using fast.com and 40Mbps using Cloudflare's tool.

[kortilla]

Your isp likely has a Netflix oca or is directly peered with Netflix directly. It’s standard for any decent sized ISP since it’s either the largest or 2nd largest by volume peer (youtube being in the other).

[egberts1]

MB = megabyte

Mb = megabits

1 byte = 8-bit

[gnicholas]

For years, AT&T sales reps would refer to MB when they meant Mb. Soon after I started service with them, I called to ask about the promised speed, and the tech insisted they sold Mb/s. He conferenced in a salesperson and was embarrassed to discover that the salesperson talked exclusively in terms of MB/s.

[NoZebra120vClip]

Did they just spell it with the abbreviations? Or did they pronounce 'megabytes' deliberately?

[gnicholas]

They literally said it out loud. I had spent the last 7 years on a fast university network, and had never paid for internet before that. I honestly believed that they were going to deliver the speeds they claimed.

[anonymousnotme]

I was trying to ask if there was something more that the speed test do other than multiple or divide by 8. Is some other overhead that they add in? If not, then their math or testing seem to be off or curl/wget seem to be different than they get via the browser's javascript engine. To me it seems that the speed test number is inflated or higher than what one will even for the same URL off of netflix or cloudflares URLs used in their speed tests.

[kijiki]

Couldn't ISPs just sniff the SNI hostname to differentiate fast.com vs actual Netflix video streaming?

[wtallis]

The fast.com page kicks off requests to nflxvideo.net domains for the actual speed measurement. And it wouldn't surprise me if actual Netflix video streaming made occasional connections to fast.com purely to make it harder for ISPs to cheat.

[cesarb]

You can think of the requests to fast.com as just loading the speed test control scripts and user interface. The actual speed test loads files from the same servers (with the same SNI hostname) used by actual Netflix video streaming. It wouldn't surprise me at all if the fast.com speed test loads real streaming video segments from these servers, the only difference being that it doesn't have the decryption key for these videos.

[bmicraft]

It would also not surprise me if dns requests for fast.com temporarily elevated bandwidth limits for netflix

[bradknowles]

It does. You're loading segments of real movies on Netflix that are not encumbered by copyright.

But they're intentionally otherwise indistinguishable from any other real movie on Netflix.

[therein]

That's surely how I would have done it.

[catminou]

Yes a few ISPs actually do this (not just for Netflix, but for other sites in general).

[sharkski]

In the case of Ookla's speedtest, many ISPs host speed test servers to eliminate any variability of the wider internet. That's why it may seem they are prioritizing it. It's not a useless tool either as lets you narrow issues down to either your network or ISP.

[londons_explore]

On my ISP (Three UK), a DNS lookup to fast.com gives you faster internet for the next minute or so.

Therefore, I have a background bash loop just looking up fast.com every 30 seconds all day long.

[thejsa]

Out of interest, which Three plan (& APN) are you on?

I’m working on using a Smarty (Three UK-owned budget label MVNO, for those unaware) SIM card in a 5G modem as a backup line, and haven’t been super impressed with speeds — so curious about trying your tip.

[bscphil]

> Since fast.com is a Netflix ip, and the isp can’t distinguish whether it’s video that is being transferred or a file to measure throughout,

It is really trivial to do basic traffic snooping and see what people are looking at. I'm surprised it isn't more common.

I figured it would be harder, or perform worse, but I easily wrote a little piece of software that filters the TLS ClientHello for arbitrary domains. Maybe 10 years ago hardware wouldn't have been able to do this, but I bet it's no big deal now. So your filter chain just looks like <Netflix IP range> -> <has fast.com ClientHello> -> unthrottle. You don't need to do packet inspection on every packet, just ones that you might be interested in (e.g. Netflix IPs).

It's crazy to me that the many people who care about privacy and censorship in tech haven't pushed ECH (encrypted ClientHello) harder. It's such a gaping hole in web privacy that you can still passively snoop domain names sent in cleartext. It makes DoH/DoT almost pointless.

[jeroenhd]

Fast.com loads from nflxvideo.net for me. They have a /speedtest/ path.

Sure, they could apply some kind of advanced DPI assessment based on packets sizes over time and other crap, but if anyone complains it's much easier to just say "well, does speedtest.net work? What about Google's speedtest? Hmm, strange, must be fast.com being slow then!". If they just check for fast.com in the SNI header, I know I'd be loading the speedtest every time I want to watch Netflix.

ECH is slowly coming along, but it's still perfectly possible to detect these speed tests. It just takes much more time and effort to set up right.

I'm honestly surprised the article shows enabling a VPN actually working to fix the messed up network here. That would be a perfectly valid solution in my opinion; if I notice my ISP is messing with my network like this, there's no way I trust them enough not to use a VPN.

[xyzzy_plugh]

This isn't applicable to fast.com as it simply makes requests to Netflix's CDN from the frontend. Indistinguishable from regular Netflix traffic.

[bscphil]

Good point, although you could still do logic like only activating the throttle after the customer visits netflix.com. You can't distinguish the CDN traffic, but you can still tell what website is being viewed.

Incidentally, my speeds on fast.com are always terrible (about 1/8 of what I get elsewhere), despite the fact that I'm fairly confident it is not being throttled. That's because the speed I see is >100 Mbps, which is like 4 Netflix UHD streams. Wouldn't be much point in throttling to that speed, you'd want 10 Mbps max, and less on wireless.

[the456gamer]

then you might miss embedded clients, right?

[hedora]

People drastically overestimate the security properties of TLS.

The correct mental model is that it’s good enough to convince 1990’s US internet users to type their credit card into a web page. (Where the downside of a breach is that you have to dispute some charges and change your CC#.)

If you need stronger security than that, then many, many caveats start to apply.

For instance, by default, anyone that can reliably man-in-the-middle port 80 on your website can get an acme certificate for your domain from a reputable certificate authority.

[snowwrestler]

If by “people,” you mean “people who rely on HTTPS for security,” then it seems like you’re saying that every hosted software company is getting it wrong. HTTPS is the foundation of the “zero trust” security model as implemented by Google, MS365, Facebook, AWS, etc.

I think it’s more likely you are not considering the full picture of the TLS ecosystem, or are making arbitrary distinctions like “cert transparency logging isn’t actually part of HTTPS” or something.

Consider that Symantec basically did what you suggest (mis-issue some certs) and not only was it detected and mitigated, they lost their CA business entirely over it.

[pizzalife]

>For instance, by default, anyone that can reliably man-in-the-middle port 80 on your website can get an acme certificate for your domain from a reputable certificate authority.

You are leaving out a huge caveat here - exactly where the MITM is taking place matters a lot. In 99% cases this isn't possible unless the victim server network is effectively compromised.

[boosteri]

Is there more on this mitm cert spoofing somewhere I can read up on? Sounds intriguing to say the least

[james-revisoai]

It probably isn't too different from other access methods: 1) Get access to port 80 for > 60 seconds. Point it at your temporary VM. 2) Use any cert authority, and for verification, choose a file-specific location verification (you can choose amongst DNS records, an email to admin@domain, or a specific file location on your site with many of them) 3) On your VM, Quickly paste the file-specific location into a django GET route. 4) Run the Django site on port 80. 5) The cert authority verifies you, and emails your account the cert, the website author being none the wiser. You can now use it later to fool future visitors for a deeper attack or email-related attacks.

[Spivak]

This doesn't work unless the attacker happens to be in between your servers and the cert authority. The ISP that's in-between your laptop and the site can't pull this trick.

Also LE actually knows this attack is possible and mitigates it by validating the challenge from multiple sources so the attacker would need to be in the middle of all the LE validators and your servers.

https://portswigger.net/daily-swig/lets-encrypt-deploys-new-...

[james-revisoai]

Yes that's true, but I was just talking about the scenario of having access to port 80 of a server DNS pointed at by some domain.

You might have access through editing a proxy rewrite rule, for example.

In the attack above you use your own SSL provider for a cert (LE not involved) and you overwrite the cert in a sense that existed before. You choose a provider that just validates with a file location. It's an attack which already requires compromise.

[snowwrestler]

> The cert authority verifies you, and emails your account the cert, the website author being none the wiser.

It’s not hard to monitor cert transparency logs, which will show this new cert.

[bradknowles]

AT&T is well know for doing deep packet inspection so that they can properly throttle NetFlix. They are always at the top of the list of Net Neutrality violators that Netflix complains about.

[eurleif]

I recently discovered that T-Mobile does this too, but they actually let you disable it on their site. Ostensibly, it's a feature for your benefit (somehow), and they're doing you a favor by enabling it by default. In reality, of course, it's for their own benefit, and they're banking on people not realizing it can be disabled. I suppose giving you the option lets them advertise things like "no throttling" and "4K streaming supported" while still reaping the benefits of throttling/lower-bitrate streaming.

[tyingq]

>it's a feature for your benefit (somehow)

They don't count the "shaped/throttled" sites against your data plan limits, so I can see some people liking it.

[eurleif]

They do this even with an unlimited plan.

[tzs]

> Ostensibly, it's a feature for your benefit (somehow), and they're doing you a favor by enabling it by default.

Some throttling by default for video actually is sort of for your benefit, at least indirectly, although I don't know if carriers are throttling more than that amount.

A surprisingly large number of people nowadays watch movies on their phones or small tablets. On screens that size at the distances they typically hold them from their eyes 4K and usually 1080p won't offer any visible improvement over 720p. On some devices even 720p probably won't offer an improvement over 480p.

It is to everyone's benefit if other people on the network don't stream at a higher resolution than is necessary to reach the limit of what is visible. Since most people aren't going to change any settings from the default, defaulting to low speed streams maximizes benefits.

[eurleif]

Speak for yourself. 1080p is noticeably worse if I'm paying attention; 720p is noticeably worse at a glance; and 480p (which is effectively T-Mobile's limit by default, even though they advertise my plan as supporting "4K streaming") looks like hot garbage.

[gr4yb34rd]

T-Mobile gives me full speed when I set my APN to B2B/IPV4.

[paulddraper]

Indeed that's the point of fast.com.... To measure the speed from Netflix servers

[kloch]

> Since fast.com is a Netflix ip

Technically the website fast.com resolves to an Akamai IP, but the speed test it loads uses a Netflix owned IP.