> "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?"
Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? No? Well, there you have it. You are not a customer, hence you aren't really in any position to demand anything. You basically agreed to take whatever Google shoves down your throat for free, and if that includes "experiments", then that's what it is.
If your multi-million dollar programs move so much money around, maybe take some of that to either invest in the necessary software - including the browser - so you are a paying customer and may actually demand anything, or pay some people to be up to date on the intricacies of Google Chrome and test them under your environment or disable them if undesired. The experiments are not exactly a secret program and have existed for a while. Firefox has a similar thing going. The Firefox one can definitely be disabled. The Chrome one unfortunately not (at least I don't know of a way at the moment, there might be one, maybe it's also only available for the Enterprise subscription that I have never personally heard of any business to be using, but for which money is actually paid, hence there might be leverage in that case to force Google to offer an opt-out).
Or ultimately you could also just compile yourself a Chromium from scratch and update it regularly. I've done it, it's not that hard, and that gets you the ultimate level of control over that nice free piece of software that you depend your business upon.
Google have made many attempts to sell Google services, and Chrome in particular[1], as professional software that businesses can rely on.
I've been at conferences where senior Google staff went to great lengths to present the benefits of switching to Chrome.
> The experiments are not exactly a secret program and have existed for a while
I've been responsible for managing Chrome in a professional environment for several years and looked in detail at many Chrome management settings. I was not aware of the experiments. They don't seem to be mentioned in the Chrome Enterprise documentation or policies.
> Both can be disabled
How can we disable them? This is on my to-do list for today, so would genuinely appreciate your help!
Whoa, I just set out to find the way to disable them. I thought it was possible, but I seem to have confused that with Firefox. Firefox calls it "studies", and you can easily opt out of them in the browser's options.
Chrome (and chromium) calls the feature "field trials" and there doesn't really seem to be a way to opt out. I'm seriously baffled by the audacity of Google here.
Seems as if the only way to actually get rid of them is to modify the Chromium codebase and compile it yourself.
It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from.
Sorry for that misinformation, I clearly have to update my original rant.
No no no. This is an isolated incident. Developers at Google are absolute Wunderkinds that know everything better than the stupid users out there, and never make any mistakes. Except this one time. So don't worry, just leave those field trials enabled, it won't happen again.
Reminds me - they still don’t allow developers to disable chrome auto-fill which renders typeahead/autocomplete functionality useless since the chrome auto fill covers up the typeahead dropdowns. https://news.ycombinator.com/item?id=21238375
The reliability of Chrome is completely astonishing - why do you want to rip the team a new one for making one honest mistake?
Any enterprise that didn't have mitigations in place cannot legitimately complain about what is probably the most reliable piece software they use (adjusted for some complexity metric).
Chrome is a generally great piece of software and I admire many of the things that Google are doing with it.
The frustration here is that Google are operating outside the way we're all used to for software updates.
If we update the software, we know to expect problems. So we can test it and (crucially) roll back.
If Google make changes outside of that then it becomes far more difficult to manage.
Yeah, the fact that Google rolled up a fix in less than two days really shows how good they software engineering is.
I am sure that those multi million dollar enterprises affected by this issue have other softwares so reliable that this was the only time they lost millions on software issue /s.
> I'm seriously baffled by the audacity of Google here
Join the club!
> It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from.
Blocking the entire hostname would cause problems with other features of Chrome. But I think blocking the particular URL would need SSL interception which isn't very appealing.
Digging in the code also brought up that URL, and it appears as if the entire feature was disabled if Chromium is built without the "Google Chrome branding" flag (and can then be reenabled by specifying a seed URL yourself via command line). Hence one might get by with building from unmodified Chromium source.
But the options to disable it on official prebuilt Chrome appear to be grim indeed. SSL interception might not even work at all, as I guess Google uses certificate pinning.
> But I think blocking the particular URL would need SSL interception which isn't very appealing.
SSL interception is very appealing in a corporate environment. Thanks for sharing this, I've raised it with the guys who run the proxies here to investigate if we can / should drop this.
It is well worth a read! There are lots of complaints from what seems to be business users of the free version, but also a few by customers of the Chrome Enterprise subscription with managed installations, and they also appear to be affected by this issue and are demanding an option to disable the trials. From that I conclude that it can be considered safe that the trials also affect paying customers, and it does indeed seem as if they also have no option to disable them, at least currently (my personal guess is this is going to change, for sure at least for the paying customers, hopefully for everyone).
Paying customers should have an option to disable it, however unpaid customers should either rollout their own solution or deal with the consequences of running business with free, uncontrolled software.
> Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome?
The idea that because you did not pay anything for a product translates into 'you have no rights' really is getting old. Even if you don't pay you have plenty of rights and the provider of the software should not - and in many places can not - walk away claiming that because you did not pay you have no rights. It may not be the rights that you want, and it may very well be that certain behavior even if you don't like it is allowed but it just simply isn't true that you have no rights.
And that contractual bit runs both ways: companies have tried to argue time and again that free users of websites and software products were bound by EULAs, that means you can't now turn around and suddenly claim that there is no contractual relationship. Yes, there is no paper contract. But you are in a relationship and if Google or any other provider claims that they have rights by the definition of the word 'consideration' so do you.
That is a straw man argument. We are not talking about an incident that may in any way be covered by the rights you might have against Google even though you did only use a free piece of software - like for example the right to not be actively and intentionally harmed by Google (think of using Chrome as trojan horse to infiltrate your network).
I am not doubting that such rights exist. But I am doubting that you have any legal right to demand Google to not enable a generally non-destructive and well-intentioned feature in some randomly chosen installations of the software that just by bad luck happens to have a bug which makes the software non-functional (but not actively harmful) in your particular terminal server environment. There clearly was no malicious intent behind this particular issue, and Google is not obliged to test every single feature on every possible combinations of systems, including every kind of remote terminal solution imaginable.
And, having established that the legal system does not help you here, I additionally doubt that you have any other leverage over Google to make them not do such feature enablements in randomly chosen installations if you don't pay for the software and if you don't happen to be a company of huge size (which might allow you to threaten to switch your hundreds of thousands of users to a different browser). But if you paid for it, Google just might be interested in keeping that cashflow flowing, and thus might be inclined to put the additional effort in to create an option for you to disable these random feature enablements, and/or to disable them outright for the paid Enterprise installations.
You wrote "You are not a customer, hence you aren't really in any position to demand anything. "
So don't go around now claiming straw man arguments. Clearly, and you seem to agree, you do have rights, and whether or not that particular right is yours remains to be seen. I can see several ways in which it just might be so that you have that right based on your expectations of performance and that a near-monopolist like Google is walking on very brittle eggs the day they start using their position to perform unsanctioned experiments on the population at large, especially when those experiments can't be opted out of.
It is a straw man because you are obviously dragging this into the realm of defending against malicious intent from Google, which clearly does not apply here. The near-monopolist argument by which you attempt to construct a "right" to demand anything more than omission of malicious behavior does not hold water: Google does not have a monopoly (or a near-monopoly) on the browser market, there are Firefox and Edge, of which both are fit for use in enterprise environments, regularly patched and compatible with practically any website out there, including even those from Google.
> an intentionally misrepresented proposition that is set up because it is easier to defeat than an opponent's real argument.
So, no, it isn’t a straw man because he’s not manipulating/misrepresenting your argument. Whether an argument is about defending malicious behavior or not has no bearing on whether or not it’s a straw man.
> We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?
-- Angry sysadmin
Well, do they pay Google for testing in their environment?
Chrome is a free-to-use product. Their rollout strategy is good. Not that they experiment in prod - the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports.
Their rollout strategy was bad. They tested it on 1% of beta users for a month. They should have ramped it to 100% in beta before thinking about prod. Maybe 1% for a week, 25% for a week, 75% for a week then 100% for a week.
Then think about enabling it in prod. Testing on a subset of the subset of people that run beta is not enough to validate the functionality.
Even then, they probably should have done a ramp in prod as well. There are certain configurations that seem unlikely to be tested in beta. VDI is one that comes to mind. Headless operations also seem less likely to be running the beta build.
> the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports.
It was in beta, had no reports, and then they turned it on for a subset of users, and created a flood of problems.
> Well what better could they do?
Release it in a new version that doesn't auto-enable for small subsets of people, but is enabled for everyone if they've deployed that version. It's how software releases used to work. Enterprise environments can then test before it gets rolled, and save this problem from occurring.
Where are all the great browser alternatives we can get paid support for? Dead.
Your claims would make sense if Google hadn't destroyed the market for paid browsers and are now trying to kill what's left of the free market too (Safari & Firefox).
Who can compete with a product which receives tens of millions of dollars of investment per year and yet is given away for $0? US competition law is a joke, all regulators are asleep at the wheel.
Exactly, and when the workaround is "just use one of the other 3 or 4 major browsers that are probably already installed on your work computer" (and if you're using an enterprise setup as mentioned in the article, you've probably got 2 or 3 things that are still locked into IE, probably IE6 for some stupid reason).
If your business is so locked into using Chrome, and specifically Chrome, and you aren't paying money to Google to ensure it keeps working as you need it, then you have only yourself to blame.
(And yes, being unable to disable the experiments thing, and not giving any warning before-hand, is a mistake on Google and the Chrome teams part, and hurts Chrome's enterprise-ready image, but that doesn't refute the point above).
If an essential piece of your business relies on something you have no control over, then you only have yourself to blame when that reliability fails and you can't fix it. Cost of doing business, at worst.
Chrome is the new IE. There are a bunch of SaaS tools and self hosted tools that only work in Chrome.
The problem with Chrome is that developers push each other to use Chrome, so they only develop for Chrome, and the stuff they make might only work in Chrome.
"experiments" is just new features that are rolled out gradually". This "experiment" was enabled on the beta branch for 5 months and no on ran into it. It was also enabled at 1% on stable and no one caught it.
If these "experiments" didn't exist, the alternative would be that the new stable version would come out and it would break everyone, so we're back to square 1.
This self-defeating attitude and Schadenfreude is puzzling, if not downright disgusting.
Google cheated their way into the #1 browser position by bundling Chrome with other installers (similar to the spyware of old), advertising for it on its other properties and misleading users into thinking that their current browser was worse.
Now that they're #1 they're hard at work implementing anti-features such as disabling an API which was used for ad filtering, trying to keep users always logged in, tracking visited URLs, etc.
Chrome is Google's trojan horse, they're not investing millions into it out of the goodness of their hearts. They're doing it because it gives them something priceless: control over the window people have to the internet.
Once the above is clear, it becomes easy to understand why Chrome auto-updates and is nearly impossible to stop from doing so.
With these experiments Google has a backdoor installed into most computers in the world.
And you're saying people should not be upset about the backdoor, because they didn't pay for the trojan horse.
It's not the people that should pay. It's Google for spying on them and for running "tests" on their computers for free.
Of course it's not that simple. There are other ways to gain leverage over Google and the Chrome development process than money. But money is the easiest one for enterprises, especially if they boast their multi-million-dollar programs, so there at least seems to be a cashflow of considerable size depending on Chrome, of which a tiny part may possibly be diverted. And for a lot of enterprises that are not exactly at Google-scale, paying for it and hence being able to request support (and maybe even make Google pay for some of the monetary damages, depending on how the contracts are written) is the only feasible option to motivate Google.
However, Chromium being open-source, there is always the possibility of taking matters into your own hand here as an alternative to the above. That doesn't come free of charge as well, of course, the money just ends up somewhere else than Google.
One thing is "well, your experiment broke our <detailed information of setup>, is there some instructions on how to disable it?" So, well, showing humbleness since this is a software that you're using for free.
The other thing is the way that was described by the article: "I am a sysadmin from a multi zillion enterprise, your idiotic experiment broke our setup. Disable this ASAP and never do something similar ever again".
These people had a voice, the problem was solved and reverted in 1.5 day. If they had paid, they would've had direct line and it potentially would've been fixed within hours instead.
I've been dealing with this bug for the last couple of days and it's been hugely frustrating.
This was not a new version of Chrome or a software update. None of our software was updated at all (and we spent a long time checking!).
But apparently Google have an ability to change a setting and globally affect the behaviour of all Chrome browsers by enabling experimental features.
We carefully manage our software updates and patching so that we can test it and roll back if it impacts the business. Google had been good at understanding "enterprise" requirements - disabling automatic updates, setting policies etc.
But this shows that they're really focused on consumers and business users will always be an afterthought.
As an admin why would you not also run the beta version of the browser in a limited setting? That would have helped you find out about this issue sooner, since the experiment was enabled there for a while.
I'm sure plenty of admins do, that doesn't mean you will test with the same flags chosen in an experiment it only means you will see changes going into stable permanently enabled a bit early.
My own experience with ChromeOs is that I switch to beta, report bugs via interfaces provided (by Google) and then the chromium team acts surprised when the regression hits stable.
>Many didn't know that Chrome engineers could run experiments on their tightly-controlled Chrome installations, let alone that Google engineers could just ship changes to everyone's browsers without any prior approval.
Uhm, isn't that the entire auto-update feature, that google ships changes without you even being aware?
We were affected and our version hadn't changed (in fact we weren't quite on the latest version - we were still testing it). We have updates disabled and are very much aware of how to manage this.
Google changed a feature flag that was automatically picked up by existing copies of Chrome and changed their behaviour.
If Chrome is critical to your company, shouldn't you be testing on Beta before it goes stable? In this case, the feature was enabled on Beta for 5 months, yet not a single impacted company caught it. Even with 1% enabled, no one caught and reported it.
This is explicit. Binary changes and flag flips are very strongly split. This is almost always a good thing, as it means you don't need to roll back binaries to revert problems.
I can see the benefits when thinking about consumers.
I think the difference here is to do with who's responsible for keeping things "working".
In a business environment there is a whole IT department who go to a lot of trouble to guarantee that their colleagues can continue working. But when Google take that control there's a clash, and it hinders the IT team's capabilities.
In the bug thread, multiple orgs stated they tried to downgrade to the last version and encountered the exact same problem. This alone is going to leave orgs circumspect about their security and Chrome’s reliability, at the very least.
My interpretation of the below quote from the article is that Google enabled the feature through pushing a configuration change to active clients. So if I'm a sysadmin and have done my due diligence and successfully tested the new build (with my configuration) before deploying it, I would be very unhappy if/when Google decides to turn this on without my consent or knowledge.
Ref. quote from article:
"Chrome engineers operate a system called Finch that lets them push updated Chrome settings to active installs, such as enabling or disabling experimental flags."
I always recommend Firefox ESR to corporate clients that use our web-based product. Long-term stability, automatic security updates, and a yearly, well-documented release cycle that allows their IT department to test the upcoming ESR release and upgrade with confidence, while we get the benefit of a modern web browser to target.
ESR user for many years, plus useragent-switcher for sites that pretend to care about my "security" (but mostly care about CSS eye candy and the latest whizzbang). Free beta tester for other people isn't in my job description ;-)
Article said they already ran it in beta and even experimented in stable!
>"The experiment/flag has been on in beta for ~5 months," said David Bienvenu, a Google Chrome engineer. "It was turned on for stable (e.g., M77, M78) via an experiment that was pushed to released Chrome Tuesday morning."
>"Prior to that, it had been on for about 1% of M77 and M78 users for a month with no reports of issues, unfortunately," he added.
The bug affected Citrix-style environments, where perhaps 10-20 users are running Chrome on the same machine.
If 1% of users are running the experiment the chance of two of those 10 users running the experiment is tiny, and when they experience the bug it's unlikely to get reported back to Google.
If something is critical infrastructure for you, better make sure you have a backup.
I've seen this in a Swiss government agency that has Firefox installed exactly for this purpose. Their main browser is the built-in one, but if it can't or must not be used (e.g. because of a zero-day threat), they can quickly roll out a change to make Firefox the default browser until the problem/threat is over.
This is the trouble with Google's aggressive auto-update policy. "Everyone is always on the latest" is mandated because everyone should have the latest security updates, but it works great except when it doesn't.
Another example is a recent issue where a Chrome update broke WebView based Android apps and stopped them from being able to make certain types of network requests. It was fixed in 2 weeks, but that 2 week period was full of unhappy customers and lost revenue.
I'm hoping the upcoming Chromium based Edge from Microsoft will allow IT Admins to control when a browser update is rolled out and give them more control over the update process.
Chrome auto-updates can be disabled for networks behind a firewall by blocking the update server address, but that's a very crude way, and doesn't allow for updating a test machine to see how the new version works, or updating to the latest minus 1 version.
This is not about auto-updates. Organizations which had disabled auto-updates were still affected. In this case Google changed Chrome behavior without changing the release version. That’s absolutely unacceptable.
With Edge being based on Chromium I expect many corporations will start switching away from Google Chrome, but it's just swapping one corporate overlord for another.
I think we are now seeing that the attitude of Google in the way they treat Chrome users is seriously worrying. There are now a litany if issues - from adversely affecting ad blockers, linking Google sync to logons to Gmail and to now rolling out “experiments” unannounced... surely by now it is time to seriously consider Firefox?
Of course, Mozilla needs to make sure they never enable extensions like the way they did for the Mr Robot one many years ago...
So when Firefox did this earlier in the year for the expired cert, everyone was freaking out that Mozilla had that ability.
Where is all the freaking out at Chrome? My theory is that is was mostly Chrome users (the majority) complaining last time when they had no personal experience.
Because for years, IT managers have been fending off requests from users to give them something other than IE6. Since 2008, Google Chrome has been the "winner" of the desktop browser wars, with IE falling away completely and Firefox's word of mouth impact plateauing against Google's relentless marketing of Chrome.
I’ve seen Chrome in so many corporate environments and never understood why. Native browsers work fine and are crafted to work reliably in the environment they’re used in. My guess is some admins simply like Chrome and thus decided everyone should. I like Firefox but I keep it at home, at work I use Microsoft Edge.
An unfortunate number of services now ask people to use them only via Chrome. In my environment, I tell them to get over themselves, and we use Firefox. But a lot of services specifically state an expectation of using Chrome. It really is the new IE6.
Not really, that would be an excuse if someone said that to you. Most enterprises use Windows Server and manage this through Group Policy, which supports all browsers for all operating systems. If you're not using Windows Server it can be more difficult.
I haven't seen this, but assuming it's true, people used to want Microsoft to follow standards, so they created the most standards-compliant browser on the market and then reject it. The answer is to reject de facto standards like IE6 and Chrome.
Google thought this feature was polished, finished, and ready for release. What used to happen was it would be pushed out in an update, and then everyone on a server would have seen the crash. This "experiment" is just slow releasing.
Can we stop the "not paying, no demanding" BS? First it's unfair to silence people just because they are not paying. Second, we are always paying. If not with money with our privacy.
The problem is the tone of voice. One thing is you asking, humbling, why did this happen, if it can be reverted since I had a problem in my specific setup and how I (as the non paying customer) can disable it. The other is to be arrogant saying that I am a multi zillion company, you're idiotic for enabling this feature and I demand you to disable it ASAP and never do this again.
The quote from this article is clearly from the later case, even if I am sure that other people reported it more humbly.
Well it wasn't just one tab. At my company all work is done on the browser (Web app) and the browser would just crash and there was no indication what could be the problem. So although it didn't cost the entirety of one a half days work per user it severely diminished productivity all around.
The article seemed to be about an enterprise where people in some job used the browser mostly to access in-house systems. That generates no revenue for Google.
I came arcoss the loading circle tab suspending on Chrome 78.0.3904.97.Even if I close the tab,the circle was still on browser,Finally I had to restore down the chrome in order to clear it up.
Its about time Google had a CFAA charge brought against them. I'm pretty sure most network admins didn't know Google had such a whopper of a backdoor deliberately snuck in through their security measures.
Even though it might break sometimes, it is one of the reasons why I choose Chrome over Firefox: The updates on Firefox take just too long (mostly before starting) on Chrome they just happen... However, it would be nice if they asked before.
> "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?"
Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? No? Well, there you have it. You are not a customer, hence you aren't really in any position to demand anything. You basically agreed to take whatever Google shoves down your throat for free, and if that includes "experiments", then that's what it is.
If your multi-million dollar programs move so much money around, maybe take some of that to either invest in the necessary software - including the browser - so you are a paying customer and may actually demand anything, or pay some people to be up to date on the intricacies of Google Chrome and test them under your environment or disable them if undesired. The experiments are not exactly a secret program and have existed for a while. Firefox has a similar thing going. The Firefox one can definitely be disabled. The Chrome one unfortunately not (at least I don't know of a way at the moment, there might be one, maybe it's also only available for the Enterprise subscription that I have never personally heard of any business to be using, but for which money is actually paid, hence there might be leverage in that case to force Google to offer an opt-out).
Or ultimately you could also just compile yourself a Chromium from scratch and update it regularly. I've done it, it's not that hard, and that gets you the ultimate level of control over that nice free piece of software that you depend your business upon.