[danpalmer]

There are a bunch of comments saying that Google is just like any other big tech company and that the exciting engineering bit has gone. My experience is only from the last 2.5 years, but I've got a slightly different take.

Engineering from >10 years ago seems like it was a wild west. Some truly stunning pieces of technology, strung together with duct tape. Everything had its own configuration language, workflow engine, monitoring solution, etc. Deployments were infrequent, unreliable, and sometimes even done from a dev's machine. I don't want to disparage engineers who worked there during that time, the systems were amazing, but everything around the edge seemed pretty disparate, and I suspect gave rise to the "promo project" style development meme.

Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID. None of these are perfect by any means, but the convergence is a really good thing. I switched team, PA, and discipline 6 months ago, and it was dead easy to get up and running because I already knew the majority of the tech, and most of that tech is pretty mature and capable.

Maybe Google has become more like other tech companies (although I doubt they have this level of convergence), but I think people glorify the old days at Google and miss that a lot of bad engineering was done. Just one example, but I suspect Google has some of the best internal security of any software company on the planet, and that's a very good thing, but it most certainly didn't have that back in the day.

[mike_hearn]

I left over ten years ago and it's hard to understand that perspective. Back when I was an SRE (~2006 to 2009) there were only one or two monitoring systems (which didn't overlap, so you could argue there was one) and a handful of config languages. Compared to anywhere else Google had military levels of discipline and order.

> Deployments were infrequent, unreliable, and sometimes even done from a dev's machine.

Deployments were weekly and done from a dev machine because that way someone was watching it and could intervene in case of unexpected problems. Some teams didn't do that and tried to automate rollouts completely. I could always tell which products weren't doing enough manual work because I'd encounter obviously broken features live in production, do a bit of digging and discover end user complaints had been piling up in the support forums for months. But nobody was reading them, and the metrics didn't show any problem, and changes flowed into prod so the team just ... didn't realize their product wasn't working. There's no substitute for trying stuff out for yourself. I encounter clearly broken software that never seems to get fixed way too often these days and I'm sure it's partly because the teams in question don't use their own product much and don't even realize anything is wrong.

[danpalmer]

I think the state of the art has moved on quite a way from this. I understand the point of view that someone should be watching a release, but the alternative is not "no one watching a release", but more that binary releases should be no-ops. With feature flagging the binary release should do nothing different so that no one watching it is not a problem.

Additionally, rolling out from a dev machine brings so many risks – security, reproducibility, human error, and so on.

I'm glad this is not the way things work anymore, and for the most part things are more reliable as a result.

[mike_hearn]

Well, to be clear "rollout from a dev machine" meant just that the rollout controller ran locally, the actual software being released was built by a release pipeline, placed into signed packages and so on. So it was all auditable. The people doing the rollouts were those who had production administrator access anyway for on-call troubleshooting and debugging and permissions were enforced, so there was no security impact. And the same process was used for flag flips so just putting everything behind flags didn't make much difference.

It doesn't sound like what's done now is a whole lot different tbh.

[skybrian]

Although there were some rough areas, Google has been considered to have pretty advanced software engineering for 20 years or more. There was plenty for a new engineer to boggle at even then.

On the other hand, someone who started in 2015 missed out on the years when Google was mostly considered to be the good guys and given the benefit of the doubt. That’s about when the culture started turning against “big tech” in general (rather than specific companies like Microsoft).

[ethbr1]

> the years when Google was mostly considered to be the good guys and given the benefit of the doubt

It's been interesting to see the tech and public zeitgeist shift on this.

Lesson: if you're going to be popular for killing a king, be careful how similarly the thing you place on your head starts to look to a crown.

[zrobotics]

Serious question: what king did Google kill? They really didn't take down a big player in an existing industry, they started by being very good at search and displaced other search companies at a time when internet search wasn't a big business.

I don't have a great answer here, but one of the things that I think caused public opinion to shift is when people started realizing how truly massive they had gotten. It's possible to avoid Facebook, Apple, Amazon, and Netflix, but by 2015 it started to become nearly impossible to avoid Google or their products.

Not sure if that's the reason public opinion shifted, I'd started to get worried about the same time I also dropped my Facebook account, so ~2011. I have successfully managed to avoid Facebook products and tracking, but I had to give up on avoiding Google, it's simply not possible unless one is willing to make extreme sacrifices. So i kinda gave up, it was starting to become digital masochism. There's a very bitter irony in me typing this on a pixel 8,but I've just accepted that there isn't any avoiding Google tracking my online life so I've just stopped caring as much as I used to.

[jauntywundrkind]

Google was the company that kept making the web better in every way. Amazing search, endless services each with solid json APIs, a fast multi-process web-browser, & investment in the web as a whole.

Google killed the king, and the king was the desktop. The king was apps. Microsoft seemed omnipotent & in total control, and the rise of the web isn't totally Google's but they sure did a lot and they sure rode that wave.

My personal feeling is that Google lost the ball in the g+ era. Up until then, it felt like Google understood their role was to help others create value, that they had to offer APIs and platforms to let other developers onto the platform, let other people expand the value proposition. G+ was an about face, a totalizing product push, and one that offered nothing to the world. Essentially no API offered. Google wanted to make g+, they wanted to run itz and if you wanted to use it, you needed to use their client and your account with them.

Where-as in the past, with efforts like Buzz, they we're trying to expand the protocols & value of the web as a whole. Once they gave up on platform & tried to be a product company, it was much harder to believe in the futures they were trying to sell.

[robertlagrant]

I don't really understand this. Don't you think search, mail, docs, maps, adsense, were products? What were the API platforms they were making up until G+?

To my recollection, G+ was actually pretty good at launch. It just was killed (or hobbled, for future killing) incredibly early for a network-effects, non-first mover product.

[ethbr1]

I'd disagree with parent a smidge and say Google turned evil when it became a platform.

When it was a disparate group of products... incentives were generally aligned with the users of those products.

When they began to look at themselves as a platform company (Google search-on-everything, Android, Chrome), that fundamentally broke and they started making sound-platform-business but user-hostile decisions.

So I guess the moral of that story is that platforms will make you rich, but you have to be very careful to enunciate your value priorities clearly to users. (E.g. Apple: "privacy"; Google: "openness"?)

[skybrian]

I don't see Google Cloud as "giving up on being a platform." It's a different kind of platform, though!

Early Google initiatives also had a high failure rate. (Buzz, for example.)

My guess is that there are still Googlers trying to improve the web. Young people are idealistic, so why wouldn't they? But nowadays it's unlikely to be successful unless it's relatively uncontroversial infrastructure. (Some examples might be things like certificate transparency and QUIC, which became HTTP/3.)

Higher-profile initiatives to really change things often fail because they raise deep suspicion and resistance. They're certain to be misinterpreted in the worst possible way.

Also, significant changes affect vested interests. Some of those vested interests are internal.

[eddyfromtheblok]

Wasn’t the G+ push a response to Facebook’s popularity, that they feared losing out on ad revenue?

[boringg]

That was circles

[justinclift]

> I think caused public opinion to shift is when people started realizing how truly massive they had gotten.

I'm personally more of the opinion that Google has caused enough serious issues for enough people - with famously no way to get the issues resolved - that they themselves seeded or caused the negative public opinion.

Combine that with Google behaviour of clearly doing things in their own interest even when not to their user's benefit (manifest v3 proposal is a good example), and many people are like "screw Google". ;)

[skybrian]

I don't think people understand what manifest v3 is trying to solve. It's a good example of how a worthwhile effort to make the Internet less terrible can be misinterpreted.

[jorams]

Manifest v3 is an effort to make a browser slightly better at its core feature, that it is importantly already fine at, at the cost of making it worse at serving the user. The internet is increasingly user-hostile, and manifest v3 makes it harder to fight back.

[reaperman]

More explanation from you on this could help. I don’t even know where to begin - everything I search just details how v3 greatly limits the efficacy of ad blockers.

[edanm]

> Serious question: what king did Google kill? They really didn't take down a big player in an existing industry, they started by being very good at search and displaced other search companies at a time when internet search wasn't a big business.

The answer is easy - Microsoft. They didn't directly take on MS's business, but in the general understanding of what was the big tech company, the one that gobbled up the best engineers, the one that startups were afraid would decide to compete with them - that was Microsoft in the late 1980s and 1990s, and it started shifting to Google.

And even though they didn't directly compete with them on being everyone's OS, they indirectly competed with them by turning the browser itself into, effectively, the OS everyone uses. They killed IE with Chrome, they made which OS you use far less relevant, they took over Mobile OS (shared with Apple, of course), and they are now competing on Cloud.

There is no question that what Google was from 2005ish to 2020ish, Microsoft was before that. You can even read pg articles about this exact thing.

[jononor]

"kill" is not a good metaphor in tech/markets, because established things and companies actually goes away. "Dethrone" is better, as the newer thing at some point becomes equal, and then dominant. But the old still exists, potentially staying in terms of absolute market value, but minority in terms of relative market value.

During the lifetime of Google: mobile has dethroned PC, video streaming has dethroned TV, SaaS over pay-once software, online advertising over physical, etc. Google has been a force in all of these (and more) - though not alone of course.

[jononor]

EDIT: because established.... rarely actually goes away

[fasa99]

google didn't so much kill a king as it maimed 6 billion peasants

[tinco]

No, that culture started turning against big tech when Microsoft and Oracle started very brazenly abusing their market domination in the late 90s.

The culture turned on Google around the same time Google lost their innocence and dropped the "Don't be evil" clause (note the dropping of that clause was not the cause, just one of the symptoms).

[ben_w]

Back in the 90s, at least in the UK, it wasn't called "big tech"; and it wasn't a major part of our direct culture as a family home might have one personal computer if they were well off but it wasn't all that important.

I suspect that even today you'd have a hard time finding a random person on the street who even knows that Oracle is a company and not a character from The Matrix or ancient Greek mythology. And if you tell them they bought Sun they'll think you mean the newspaper, if you talk about Java they'll either think you mean the island or are talking about a brand of coffee.

[thaumasiotes]

> around the same time Google lost their innocence and dropped the "Don't be evil" clause

When did that happen? It's still there.

[pfsalter]

Back in 2018 [1], at a similar time that there was a lot of moving about and restructuring. I think this is about the time that Google search started going downhill as well

[1] https://gizmodo.com/google-removes-nearly-all-mentions-of-do...

[thaumasiotes]

Again, the phrase was not dropped. You might notice that your link is titled "Google removes nearly all mentions of don't be evil from its code of conduct". (By which they mean 3 mentions.)

But as that article notes, they didn't drop the phrase. It's there now. It's always been there. There was never a time when they dropped it.

[throwaway20222]

From my experience crossing eras; the tools have improved, and there are still some amazingly brilliant people, but the amount of overhead and the ratio of top level, passionate builders has dropped meaningfully. I have had a priority project stuck in legal hell for six months because it’s just a morass of logistics. The project is benign but it doesn’t matter. We are ossified.

I look at it as somewhat inevitable considering the path the company has taken, but it is certainly different. We are cranking out money and that’s fine, but it is a change.

[xnx]

Success is its own trap. At a certain size, there's more to lose than to gain with any change.

[whiterknight]

"Bad engineering" in adhoc ways tends to mean new ideas are being explored. Sophisticated deployments, logistics, procedures, tends to mean you're optimizing or extending existing system.

That's not to disparage the latter. making things work at scale is hard engineering. But when people praise the glory days, it may be a preference for working on new ideas in small projects.

[danpalmer]

There are most definitely trade-offs. Getting a small project off the ground now is hard because the size of the organisation puts so many requirements on you.

The way I like to look at it is that small things take a while, but big things happen remarkably quickly. For example, rolling out a "hello world" service might take a few days, but having that service serve 1M QPS is pretty much free (in terms of effort). At my previous place a new service might have taken an hour to set up a deployment for, but having it serve 1M QPS would have required overhauling several aspects of our infrastructure over months.

[Rebelgecko]

On the flip side, clueless execs are killing tools like Code Search because they don't understand the value. They're willing to layoff off a team to save money even if it reduces the productivity of 20,000 employees by 1%

[tarsinge]

What’s the point if the capacity to deploy good products has been lost in the process? What if what you see as unreliable was at the hearth of what enabled great things to happen. This is a typical caveat of a programmer centric organization. Things get more reliable but more frozen and innovation die.

[miki123211]

> Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID

Could you elaborate on what these are for a non-googler? Ironically enough, Google isn't very helpful.

[aoeusnth1]

Boq/Pod: canonical service frameworks and configuration automation all-in-one system. Boq and Pod give you blueprints for server configs, deployment, server discovery, environments + release pipelines, monitoring, alerting, canary analysis, functional testing, integration testing, unit testing and presubmit, server throttling, etc etc all for free with automated setup.

P2020 + Rollouts: This is for intent-driven deployment, where deployment configuration for jobs, networks, pubsub topics, and other resources are declaratively checked into source, and the Annealing system automatically finds diffs against production state and resolves those diffs (aka a rollout).

Automated job sizing: load-trend driven automated capacity planning. Separate from autopilot, which is a more time-sensitive autoscaler. This will actually make configuration changes based on trends in traffic, and request quota for you automatically (with your approval).

BCID: this is for verifiable binaries and configs in prod, where it requires two parties to make source changes, two parties to make config changes, two parties to approve non-automated production changes, and only verified check-in binaries and configs can run in prod, not stuff you build on your desktop machine.

[munksbeer]

> Boq/Pod: canonical service frameworks and configuration automation all-in-one system. Boq and Pod give you blueprints for server configs, deployment, server discovery, environments + release pipelines, monitoring, alerting, canary analysis, functional testing, integration testing, unit testing and presubmit, server throttling, etc etc all for free with automated setup.

That sounds amazing.

[nazcan]

These are well-lit paths, that try to make it easy to take a server you want to run in production, and give you reasonable releases/monitoring/canarying for free (e.g. no one should have to configure something that stops pushes that are causing crashes of the updates tasks).

[QuercusMax]

And even more importantly, the teams responsible have (at least in the past) done a pretty decent job of keeping things up-to-date and automatically moving you onto newer versions of systems. My services on Boq/Pod have been moved to new rollout-management platforms multiple times and I didn't have to do anything besides learn the new UIs.

[mathteddybear]

can't speak for others, but BCID is this thingy: maybe your team has some petabytes quota in the datacenter, but as a software engineer, you no longer can test your number-crunching data-processing program by running it there, not before checking it into repository. Instead, you'll have to check it in, and then run, and then check in the fixes you found

[contrarian1234]

So sure they can very reliably and safely deploy... It just doesn't feel like they've actually deployed anything interesting in 5+ years (except maybe some AI stuff). From the outside it looks like the company is perpetually refactoring

history's biggest yakshave...? not surprising for a programmer-run company

I'm sure it still matters to bring down the power and server bills. But one can't help feel like they could be doing much more

[hiddencost]

Oof. I knew it was going to happen, but people thinking boq/pod are good rather than actually killing the company makes me pretty sad.

[alienchow]

Boq/PoD merely standardizes production deployments. You can choose to not do it, and end up having to redo everything yourself. The vast majority of all services uses it perfectly fine.

Unified rollouts, unified versioning, universal dashboards, security compliance, standardized quota management, standard alert configs. It's opinionated, but I can drop into any team and hit the ground running. I don't want to learn your custom dashboards doing the exact same thing with different names.

The issue with PoD is that it's a great concept and implementation that's tight on resources, and doesn't have much of a plan to expand beyond its current paradigm. The P2020 team deserves way more recognition for all the work they have done.

[elevatedastalt]

Please tell me how you think Boq is killing the company.

[aorloff]

> 10 years ago if you were operating at scale you still racked your own servers