[Slartie]

I chuckled at this:

> "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?"

Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? No? Well, there you have it. You are not a customer, hence you aren't really in any position to demand anything. You basically agreed to take whatever Google shoves down your throat for free, and if that includes "experiments", then that's what it is.

If your multi-million dollar programs move so much money around, maybe take some of that to either invest in the necessary software - including the browser - so you are a paying customer and may actually demand anything, or pay some people to be up to date on the intricacies of Google Chrome and test them under your environment or disable them if undesired. The experiments are not exactly a secret program and have existed for a while. Firefox has a similar thing going. The Firefox one can definitely be disabled. The Chrome one unfortunately not (at least I don't know of a way at the moment, there might be one, maybe it's also only available for the Enterprise subscription that I have never personally heard of any business to be using, but for which money is actually paid, hence there might be leverage in that case to force Google to offer an opt-out).

Or ultimately you could also just compile yourself a Chromium from scratch and update it regularly. I've done it, it's not that hard, and that gets you the ultimate level of control over that nice free piece of software that you depend your business upon.

[omh]

Google have made many attempts to sell Google services, and Chrome in particular[1], as professional software that businesses can rely on.

I've been at conferences where senior Google staff went to great lengths to present the benefits of switching to Chrome.

> The experiments are not exactly a secret program and have existed for a while

I've been responsible for managing Chrome in a professional environment for several years and looked in detail at many Chrome management settings. I was not aware of the experiments. They don't seem to be mentioned in the Chrome Enterprise documentation or policies.

> Both can be disabled

How can we disable them? This is on my to-do list for today, so would genuinely appreciate your help!

[1] https://cloud.google.com/chrome-enterprise/

[Slartie]

Whoa, I just set out to find the way to disable them. I thought it was possible, but I seem to have confused that with Firefox. Firefox calls it "studies", and you can easily opt out of them in the browser's options.

Chrome (and chromium) calls the feature "field trials" and there doesn't really seem to be a way to opt out. I'm seriously baffled by the audacity of Google here.

Seems as if the only way to actually get rid of them is to modify the Chromium codebase and compile it yourself.

It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from.

Sorry for that misinformation, I clearly have to update my original rant.

[fps_doug]

No no no. This is an isolated incident. Developers at Google are absolute Wunderkinds that know everything better than the stupid users out there, and never make any mistakes. Except this one time. So don't worry, just leave those field trials enabled, it won't happen again.

[obituary_latte]

Reminds me - they still don’t allow developers to disable chrome auto-fill which renders typeahead/autocomplete functionality useless since the chrome auto fill covers up the typeahead dropdowns. https://news.ycombinator.com/item?id=21238375

[robocat]

The reliability of Chrome is completely astonishing - why do you want to rip the team a new one for making one honest mistake?

Any enterprise that didn't have mitigations in place cannot legitimately complain about what is probably the most reliable piece software they use (adjusted for some complexity metric).

[omh]

Chrome is a generally great piece of software and I admire many of the things that Google are doing with it.

The frustration here is that Google are operating outside the way we're all used to for software updates. If we update the software, we know to expect problems. So we can test it and (crucially) roll back.

If Google make changes outside of that then it becomes far more difficult to manage.

[m45t3r]

Yeah, the fact that Google rolled up a fix in less than two days really shows how good they software engineering is.

I am sure that those multi million dollar enterprises affected by this issue have other softwares so reliable that this was the only time they lost millions on software issue /s.

[omh]

> I'm seriously baffled by the audacity of Google here

Join the club!

> It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from.

It looks like they're downloaded from https://clients4.google.com/chrome-variations/seed

Blocking the entire hostname would cause problems with other features of Chrome. But I think blocking the particular URL would need SSL interception which isn't very appealing.

[Slartie]

Digging in the code also brought up that URL, and it appears as if the entire feature was disabled if Chromium is built without the "Google Chrome branding" flag (and can then be reenabled by specifying a seed URL yourself via command line). Hence one might get by with building from unmodified Chromium source.

But the options to disable it on official prebuilt Chrome appear to be grim indeed. SSL interception might not even work at all, as I guess Google uses certificate pinning.

[technofiend]

Amusingly Chromium shows up in my list of things to disable in the Google security checks results.

[oarsinsync]

> But I think blocking the particular URL would need SSL interception which isn't very appealing.

SSL interception is very appealing in a corporate environment. Thanks for sharing this, I've raised it with the guys who run the proxies here to investigate if we can / should drop this.

[patrec]

Is just patching the url in the binary an option?

[Slartie]

I found this Chromium bug report, which seems to be the epicenter of all the debate and also the source of the citations from the linked article:

https://bugs.chromium.org/p/chromium/issues/detail?id=102483...

It is well worth a read! There are lots of complaints from what seems to be business users of the free version, but also a few by customers of the Chrome Enterprise subscription with managed installations, and they also appear to be affected by this issue and are demanding an option to disable the trials. From that I conclude that it can be considered safe that the trials also affect paying customers, and it does indeed seem as if they also have no option to disable them, at least currently (my personal guess is this is going to change, for sure at least for the paying customers, hopefully for everyone).

[m45t3r]

Paying customers should have an option to disable it, however unpaid customers should either rollout their own solution or deal with the consequences of running business with free, uncontrolled software.

[jacquesm]

> Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome?

The idea that because you did not pay anything for a product translates into 'you have no rights' really is getting old. Even if you don't pay you have plenty of rights and the provider of the software should not - and in many places can not - walk away claiming that because you did not pay you have no rights. It may not be the rights that you want, and it may very well be that certain behavior even if you don't like it is allowed but it just simply isn't true that you have no rights.

And that contractual bit runs both ways: companies have tried to argue time and again that free users of websites and software products were bound by EULAs, that means you can't now turn around and suddenly claim that there is no contractual relationship. Yes, there is no paper contract. But you are in a relationship and if Google or any other provider claims that they have rights by the definition of the word 'consideration' so do you.

[Slartie]

That is a straw man argument. We are not talking about an incident that may in any way be covered by the rights you might have against Google even though you did only use a free piece of software - like for example the right to not be actively and intentionally harmed by Google (think of using Chrome as trojan horse to infiltrate your network).

I am not doubting that such rights exist. But I am doubting that you have any legal right to demand Google to not enable a generally non-destructive and well-intentioned feature in some randomly chosen installations of the software that just by bad luck happens to have a bug which makes the software non-functional (but not actively harmful) in your particular terminal server environment. There clearly was no malicious intent behind this particular issue, and Google is not obliged to test every single feature on every possible combinations of systems, including every kind of remote terminal solution imaginable.

And, having established that the legal system does not help you here, I additionally doubt that you have any other leverage over Google to make them not do such feature enablements in randomly chosen installations if you don't pay for the software and if you don't happen to be a company of huge size (which might allow you to threaten to switch your hundreds of thousands of users to a different browser). But if you paid for it, Google just might be interested in keeping that cashflow flowing, and thus might be inclined to put the additional effort in to create an option for you to disable these random feature enablements, and/or to disable them outright for the paid Enterprise installations.

[jacquesm]

You wrote "You are not a customer, hence you aren't really in any position to demand anything. "

So don't go around now claiming straw man arguments. Clearly, and you seem to agree, you do have rights, and whether or not that particular right is yours remains to be seen. I can see several ways in which it just might be so that you have that right based on your expectations of performance and that a near-monopolist like Google is walking on very brittle eggs the day they start using their position to perform unsanctioned experiments on the population at large, especially when those experiments can't be opted out of.

[Slartie]

It is a straw man because you are obviously dragging this into the realm of defending against malicious intent from Google, which clearly does not apply here. The near-monopolist argument by which you attempt to construct a "right" to demand anything more than omission of malicious behavior does not hold water: Google does not have a monopoly (or a near-monopoly) on the browser market, there are Firefox and Edge, of which both are fit for use in enterprise environments, regularly patched and compatible with practically any website out there, including even those from Google.

[colejohnson66]

From the dictionary:

> straw man (n):

> an intentionally misrepresented proposition that is set up because it is easier to defeat than an opponent's real argument.

So, no, it isn’t a straw man because he’s not manipulating/misrepresenting your argument. Whether an argument is about defending malicious behavior or not has no bearing on whether or not it’s a straw man.

[Slartie]

Moving the argument, which has been about a user of a free software program having neither an actual nor a moral capability of demanding that the free software fits their particular needs (which in this case means to work as a critical part of their business and continue working as such under their particular environment) into the legal realm and making it into a proposition that a user of a free software has no rights at all, not even legal rights to defend against malicious behavior of the producer of the software, is an intentional misrepresentation of the argument, and it is easier to defeat than the actual argument since there are clearly such defensive rights.

[jve]

Yeah.

> We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'? -- Angry sysadmin

Well, do they pay Google for testing in their environment?

Chrome is a free-to-use product. Their rollout strategy is good. Not that they experiment in prod - the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports.

Well what better could they do?

[curryst]

> Their rollout strategy is good.

Their rollout strategy was bad. They tested it on 1% of beta users for a month. They should have ramped it to 100% in beta before thinking about prod. Maybe 1% for a week, 25% for a week, 75% for a week then 100% for a week.

Then think about enabling it in prod. Testing on a subset of the subset of people that run beta is not enough to validate the functionality.

Even then, they probably should have done a ramp in prod as well. There are certain configurations that seem unlikely to be tested in beta. VDI is one that comes to mind. Headless operations also seem less likely to be running the beta build.

[oarsinsync]

> Their rollout strategy is good.

I think you've misunderstood what they did

> the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports.

It was in beta, had no reports, and then they turned it on for a subset of users, and created a flood of problems.

> Well what better could they do?

Release it in a new version that doesn't auto-enable for small subsets of people, but is enabled for everyone if they've deployed that version. It's how software releases used to work. Enterprise environments can then test before it gets rolled, and save this problem from occurring.

[blub]

Where are all the great browser alternatives we can get paid support for? Dead.

Your claims would make sense if Google hadn't destroyed the market for paid browsers and are now trying to kill what's left of the free market too (Safari & Firefox).

Who can compete with a product which receives tens of millions of dollars of investment per year and yet is given away for $0? US competition law is a joke, all regulators are asleep at the wheel.

[vanadium]

Making experiments opt-in seems like a good first step, no? As it is, there wasn’t even a clear opt-OUT.

[jacquesm]

Was there any opt-out at all? Even an unclear one would do, then at least users can point each other to the option.

[davvolun]

Exactly, and when the workaround is "just use one of the other 3 or 4 major browsers that are probably already installed on your work computer" (and if you're using an enterprise setup as mentioned in the article, you've probably got 2 or 3 things that are still locked into IE, probably IE6 for some stupid reason).

If your business is so locked into using Chrome, and specifically Chrome, and you aren't paying money to Google to ensure it keeps working as you need it, then you have only yourself to blame.

(And yes, being unable to disable the experiments thing, and not giving any warning before-hand, is a mistake on Google and the Chrome teams part, and hurts Chrome's enterprise-ready image, but that doesn't refute the point above).

If an essential piece of your business relies on something you have no control over, then you only have yourself to blame when that reliability fails and you can't fix it. Cost of doing business, at worst.

[lykr0n]

Chrome is the new IE. There are a bunch of SaaS tools and self hosted tools that only work in Chrome.

The problem with Chrome is that developers push each other to use Chrome, so they only develop for Chrome, and the stuff they make might only work in Chrome.

[davvolun]

Absolutely, and if your business is repeating the same mistakes in Chrome as we saw 10-20 years ago from IE....

[tyingq]

Does Google run these experiments on the $50/year per device Chrome Enterprise?

[ehsankia]

"experiments" is just new features that are rolled out gradually". This "experiment" was enabled on the beta branch for 5 months and no on ran into it. It was also enabled at 1% on stable and no one caught it.

If these "experiments" didn't exist, the alternative would be that the new stable version would come out and it would break everyone, so we're back to square 1.

[blub]

This self-defeating attitude and Schadenfreude is puzzling, if not downright disgusting.

Google cheated their way into the #1 browser position by bundling Chrome with other installers (similar to the spyware of old), advertising for it on its other properties and misleading users into thinking that their current browser was worse.

Now that they're #1 they're hard at work implementing anti-features such as disabling an API which was used for ad filtering, trying to keep users always logged in, tracking visited URLs, etc.

Chrome is Google's trojan horse, they're not investing millions into it out of the goodness of their hearts. They're doing it because it gives them something priceless: control over the window people have to the internet.

Once the above is clear, it becomes easy to understand why Chrome auto-updates and is nearly impossible to stop from doing so. With these experiments Google has a backdoor installed into most computers in the world.

And you're saying people should not be upset about the backdoor, because they didn't pay for the trojan horse.

It's not the people that should pay. It's Google for spying on them and for running "tests" on their computers for free.

[chris_wot]

Some of them do. If a company purchases GSuite, GSuite for Education or Chrome Enterprise Support then they have paid for this.

https://support.google.com/chrome/a/answer/188447?hl=en

[m-p-3]

Does being a GSuite customer counts as paying for Google Chrome?

If that doesn't count, there there is no scenario where you can say you've paid for it.

[chris_wot]

It does. Under “Can I get phone support” it states:

Yes, if you have a G Suite or G Suite for Education account, or if your company purchased Chrome Enterprise Support.

https://support.google.com/chrome/a/answer/188447?hl=en

[user740372]

So no pay, no voice?

[Slartie]

Of course it's not that simple. There are other ways to gain leverage over Google and the Chrome development process than money. But money is the easiest one for enterprises, especially if they boast their multi-million-dollar programs, so there at least seems to be a cashflow of considerable size depending on Chrome, of which a tiny part may possibly be diverted. And for a lot of enterprises that are not exactly at Google-scale, paying for it and hence being able to request support (and maybe even make Google pay for some of the monetary damages, depending on how the contracts are written) is the only feasible option to motivate Google.

However, Chromium being open-source, there is always the possibility of taking matters into your own hand here as an alternative to the above. That doesn't come free of charge as well, of course, the money just ends up somewhere else than Google.

[m45t3r]

One thing is "well, your experiment broke our <detailed information of setup>, is there some instructions on how to disable it?" So, well, showing humbleness since this is a software that you're using for free.

The other thing is the way that was described by the article: "I am a sysadmin from a multi zillion enterprise, your idiotic experiment broke our setup. Disable this ASAP and never do something similar ever again".

[ehsankia]

These people had a voice, the problem was solved and reverted in 1.5 day. If they had paid, they would've had direct line and it potentially would've been fixed within hours instead.