[sethbannon]

The overarching message is that builders should deeply consider the impact of what they're building on civilization.

"Technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it."

Therefore builders "bear a particular ethical and spiritual responsibility" because "every design choice reflects a vision of humanity."

The questions shouldn't just be 'can we build it?' or 'will people want this?'

We need to also ask 'should we build it?' and 'will this make humanity better?'

The encyclical calls on us to “join forces in building up the common good.”

This is a message we need right now.

[ThrowawayR2]

Not new for software and hardware industry though, practitioners have just chosen to ignore it. From the Association for Computing Machinery, which encompasses all forms of software development, the very first principle is the public good:

"Software engineers shall act consistently with the public interest. In particular, software engineers shall, as appropriate:

1.01. Accept full responsibility for their own work.

1.02. Moderate the interests of the software engineer, the employer, the client and the users with the public good.

1.03. Approve software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good. ..."

From the IEEE, which also encompasses computer engineering, their first principle and its first few sub-items are:

"To uphold the highest standards of integrity, responsible behavior, and ethical conduct in professional activities.

1. to hold paramount the safety, health, and welfare of the public, to strive to comply with ethical design and sustainable development practices, to protect the privacy of others, and to disclose promptly factors that might endanger the public or the environment;

2. to improve the understanding by individuals and society of the capabilities and societal implications of conventional and emerging technologies, including intelligent systems; ..."

[hnthrow0287345]

This would be more meaningful if, perhaps, we had to swear an oath to it before being able to practice. And practitioners would be treated more seriously if everyone knew we swore that oath. And the legal utility as accountability and defense would also be useful.

Of course people are going to ignore it if there's no force behind it.

[opsnooperfax]

Professional liability and licensure would create assurances with some teeth, but there are some major drawbacks.

[gymbeaux]

Enlighten us to these drawbacks. On the surface I am inclined to say the pros would outweigh the cons. Compared to other professions, software engineering seems to struggle the most with H-1B/Green Card abuse and interview processes. Job interviews are absurdly different (easier) for doctors, lawyers, et al. than for software engineers, and that I believe is because of the licensure. I do think licensure adds overhead to an industry (e.g., malpractice insurance, governing bodies, license management) and that probably discourages anyone with real power (like FAANG) to pursue it and try to set it as an industry-wide standard. Most software engineers in the U.S. are making around $130-140k, but lawyers and medical doctors usually make significantly more (perhaps because of the licensure overhead - I'm not sure if malpractice insurance is included in a medical doctor's salary- I would imagine it's not and is taken out of each paycheck like any other industry's health insurance benefits).

[ThrowawayR2]

> "Job interviews are absurdly different (easier) for doctors, ..."

The job interview for doctors is a 5-7 year residency under tight supervision of an attending physician: https://en.wikipedia.org/wiki/Residency_(medicine) . People do flunk or drop out as well, meaning they can never become a physician. There's nothing easier about that.

[raphman]

I think the parent commenter agrees with you: because there is tight quality assurance and - in many countries - a license needed to practice medicine, the interviewer can just trust the system instead of having to evaluate the competence of the applicant through questions and coding assignments.

(I'm not sure whether I agree with the commentator that a SE license would be that helpful in practice.)

[gymbeaux]

I’ve been a software engineer for over a decade. At what point is my “residency” over? At what point does the “community” decide I can write software with some level of competency and don’t need to solve fizz buzz over and over again?

[lukan]

It seems to work somewhat with medical doctors and the Hippocratic Oath.

But I would argue it is way easier there. Building software has way more grey areas.

[trolleski]

I don't think there are more grey areas in software engineering than in medicine. The difference is the feedback loop of the outcome - if you design a dopamine slot machine you will ruin the generation and that's a long arc.

[lukan]

And that makes it hard. I am open for banning all comercial advertisement - but general society is largely fine with it. So is someone designing new targeting algorithm for ads breaking his potential oath of doing good for society?

What grey areas are there for doctors?

[trolleski]

Prescribing meds which you could avoid while receiving 'courses' from Key Opinion Leaders in fancy places from the pharma companies. This ranges from antibiotics through psychotropics and up until the pain meds. You can argue some of it is on the safe side, some may be beneficial, etc. Should you accept an invitation to a conference which will increase your skills?

[boppo1]

Tangential, but where do you draw the line on commercial advertisement? If a podcast is sponsored by a business & supports behavior the business benefits from, is the podcast advertising? What if it also contains useful educational content related to that activity?

[brcmthrowaway]

Canadians do

[jimbokun]

We need to find an organization to which software engineers can report corporations abusing these principles, that can then take legal action or at least disclose those practices to the general public.

[MichaelZuo]

Noble ideals can be upheld, relatively consistently… only if violators are visibly punished. And to a degree roughly commensurate with the violation.

The critical point is that the violators have to be punished more consistently then the demanded consistency of the ideals.

[mentalgear]

I think not just (computer) scientists but the general population thinks to serve the common good makes sense, not last because we understand it's eventually for our own good.

It is however just that very small minority of the population with highly psychopathic/narcissistic traits - those that, in pre-historic times, would have been kicked out swiftly of hunter & gatherer / small village communities because of their parasitic nature - that in bigger civilisations seem to thrive due to abstraction (distance/time of the effect of their actions) and obfuscation (PR) and instead unfortunately seem to rise to the top (CEOs, presidents, 'thought leaders' ...) to steer the world's overall economy and mindset - and steer it in the abyss.

Sometimes I think humanity was just not made to scale, and this aspect is one very large aspect of it.

[aerodexis]

Humanity was not made to scale quickly. The trouble we have nowadays is that there is no backpressure, largely because the systems that generated that backpressure were dismantled in the name of "freedom" and other myths.

[RugnirViking]

I agree. Heck, it was only 20 or 30 years ago that a common exhortation was to not "sell out". Nowadays that is gone, replaced only by "get that bread". We need to hold each other to a higher standard

[jimbokun]

You are referencing prehistoric times because there is no evidence to confirm or refute your claims.

Most likely the same kind of Machiavellian political maneuvering was effective in those groups as well.

[vermilingua]

Not only builders, the greatest takeaway for me is that everyone has a responsibility in shaping the discourse, culture, and usage of transformative technology. This "the builders will do the right thing" mentality is even (in my interpretation) explicitly called out in several places:

> It is the pursuit of the common good that gives life to a people, understood not as a mere collection of individuals, but as a living reality in which people learn to recognize that they themselves are interconnected and jointly responsible for the res publica. In this sense, every person contributes to the building up of one’s people...

> When it comes to decisions regarding economic flows and digital platforms, as well as the governance of data and algorithms, we cannot allow a handful of actors to dictate these processes on their own; instead, we must build forms of cooperation that respect the various levels of the global community and make them jointly responsible for the common good.

> We cannot be satisfied with merely calling for the moralization of machines — the so-called “alignment” of AI with human values — without also having the courage to insist on a further condition: the possibility of openly discussing the ethical frameworks involved and subjecting them to shared standards of social justice.... What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.

[ako]

Agreed, I think the builders are the wrong people to ask for this self reflection. Anything can be used for good and bad. A knife can be used to prepare food, or to stab someone. A drone can be used to attack a country, or to defend a country. A wooden beam can be used to build a house, or to build a cross to crucify someone. Same way AI can be used for good and bad. And it’s up to the person using the tool to act moral. Unfortunately this world is full of people that consider power and money more important than morals. For me, religion falls in this bucket, lots of “do as I say, not do as I do”. And on top of this, there’s very little agreement on what is good, and what is bad. Morals are quite fuzzy, and flexible.

[jimbokun]

You are welcome to your opinion, but human civilization was even more biased towards instrumentality before the broad spread of Christianity.

I am well aware of the abuses and scandals of Christian churches throughout history. I am also aware that the Romans and other civilizations pre-dating Christianity had even less regard for the poor and powerless.

[hammock]

Yes. I like WEF and Klaus Schwab’s stakeholder capitalism model here. We need governments to force companies to answer not just to shareholders but to employees and communities, and we need one world government to force individual governments to do that.

[devilbunny]

The Roman Catholic Church has a lot of things wrong with it, now and in the past, but it’s a human institution older than almost any other, and it’s composed of a lot of very intelligent people. Agree or disagree with them, but a papal encyclical is almost always worth reading and understanding.

[ycombinete]

Indeed. And because of it’s age and outlook its view is very far back and very far forward.

The casual reference to a 135 year old encyclical that dealt with the seismic shift of industrialisation took me quite aback for a number of reasons.

[michaelsbradley]

There are so many encyclicals, apostolic letters, etc. One could spend years reading just a fraction of them, depending on reading and comprehension speed, of course, which varies by person.

Two I recommend, from the last 40 years:

Veritatis splendor, John Paul II, 1993

Argues that Christian freedom is fulfilled, not limited, by objective moral truth: some acts are intrinsically evil regardless of intention or circumstance, conscience must be formed by divine law rather than self-authorization, and the Church must faithfully teach this moral truth as the path to authentic human flourishing in Christ.

https://www.vatican.va/content/john-paul-ii/en/encyclicals/d...

Fides et Ratio, John Paul II, 1998

Argues that faith and reason are complementary paths to truth: reason needs faith to avoid skepticism, relativism, and reductionism, while faith needs reason to express, defend, and deepen its understanding of divine revelation and the human search for meaning.

https://www.vatican.va/content/john-paul-ii/en/encyclicals/d...

[thuridas]

Note that, as anybody, the Catholic Church may have is bias.

For example. Their dualist view of the world makes them see AI as something very different from human intelligence. So, without having read it, the church may negate that it could be at human level. A few years ago that would negate that computers could be creative because they do not have a soul.

Anyway, kudos for focussing on the important issues and the impact on human.(And less on sex)

[aerodexis]

I have yet to read the whole thing - but I agree that the Church's view of intelligence is not to the level of sophistication needed to counter the Valley's pantheistic view of intelligence. I think that is because of how Aquinas was utilized to counter the Reformation. That being said, I don't believe that such a view cannot be elaborated, it will just take time. The key is embodiment, wherein how we view sex ends up being incredibly important because it necessarily relates to how we take on flesh to begin with. Once sex is divorced from procreation (and vice-versa), intelligence is divorced from humanity. It's very relevant - but the culture of dehumanization is so deeply rooted today that it's difficult to be productive when tackling that dehumanization via sex.

[twitchard]

Fwiw the teaching of the church is that (today's) AI isn't at a human level because human intelligence is something that we experience whereas artificial intelligence seems to be merely a sophistication in performing tasks.

This week's encyclical didn't go into this. Last year's "Doctrinal Note" (less authoritative) did though: https://www.vatican.va/roman_curia/congregations/cfaith/docu...

and I did a write-up on it here: https://twitchard.github.io/posts/2025-06-28-the-catholic-ch...

[cma]

I would also recommend God and Golem, Inc., A Comment on Certain Points Where Cybernetics Impinges on Religion:

https://monoskop.org/images/1/1f/Wiener_Norbert_God_and_Gole...

Norbert Weiner was an atheist but he talks about three areas religion is the only thing to have really examined that relate to capable AI: omniscience, omnipotence, and worship (gadget worship). It has very prescient stuff on blackbox learning/distillation, reinforcement learning/reward hacking, alignment through human feedback.

His The Human Use of Human beings and Cybernetics are extremely good too and have more of a mash of the themes between Rerum Novarum and Magnifca Humanitas, and more near-term automation.

[runlaszlorun]

Awesome, thx for posting. I now have my new next book to read. Been wanting to read more of the original cybernetics stuff.

Would you also recommend "The Human Use of Human Beings"?

[cma]

Yes, it starts slow with a lot of history of Cybernetics stuff through Leibniz and stuff (kind of prescient given chain rule -> back propagation, and control theory's relevance to optimizers). It is about twice as long as God & Golem and covers a lot of the same plus more examination of automation and human augmentation. I think either it or Cybernetics also goes into mass communication with some relevance to how social media played out.

God & Golem is the most succinct and up to date though, probably a 2hr read.

The book Cybernetics is a lot of math and ergodic theory stuff that went beyond me, but is the longest and you still get a lot out of it skimming over that stuff if you don't have the background for it. The last revision of it in the 50s added some of the same blackbox function copying/imitation learning/distillation stuff, reinforcement learning with reward hacking concerns, and superintelligence as genie/monkey's paw.

I would read the three in reverse order of publication.

He also foresaw another big area of potential existential danger, Wiener filter for guidance and control of missiles (later superceded with Kalman filter bringing the nuclear hard targets era with 15min retaliation windows) and refused to work on it or share prior work, and he also had bioweapons delivery concerns before the bioweapons treaties, publishing this open letter in The Atlantic in 1947:

https://archive.ph/D7BPt

[davidivadavid]

That book cover goes so hard.

[ctoth]

The strange part is how moral responsibility somehow always lands on the builders... the people with the least leverage... while the funders get to ask the ethical questions. Weird!

[ThrowawayR2]

No, we don't have to take the funders' money; that's what having professional standards means. Nobody would excuse a doctor performing unsafe procedures because they "needed the money". Engineers were jailed for the Volkswagen emissions tampering scandal and nobody would excuse them for needing to take funders' money.

[erikerikson]

I agree that holding only the concrete implementers responsible would be inappropriate. However, I don't believe that distinction is made. One says "I am building a house" even if they are completely contracting the job out. I'd suggest the greatest responsibility lays with the funders and that the Pope would agree.

[jeremyjh]

The funders are among the principal builders in this context. This is addressing the people who have a say in what is built, and how it is built. Much of that belongs to the executive and ownership class, but not all of it.

[arter45]

It lands on both. "It was just a job" or "I was just following orders" doesn't excuse you from doing unethical stuff.

[alok-g]

Who do you consider to be by-and-large, overall, more intelligent? Who then is more capable of exercising intelligence towards figuring out what's actually good?

[hodgesrm]

Mark Zuckerberg is a builder in the context of the encyclical.

[jimbokun]

I’m only through the introduction but this encyclical makes clear everyone bears some responsibility to act, including but not limited to the builders.

[aerodexis]

Just following orders huh...

[lishzen]

Users are responsible, too

[Xenoamorphous]

> Scientists are actually preoccupied with accomplishment. So they are focused on whether they can do something. They never stop to ask if they should do something.

From Crichton's book Jurassic Park, which like most of his books is about the perils of technological advancements.

They used the quote in the movie, slightly tweaked.

[throw4847285]

I would replace the word scientists with engineers in that quote. People often conflate the two, but in my experience, scientists tend to be more cautious and there are built in checks and balances in the process (however flawed).

Engineers/technologists tend to have no such guardrails, and are also usually embedded into entirely profit motivated environments, whatever their own values might be.

[velcrovan]

This sounds like you don’t have much exposure to actual professional engineering disciplines. I’m sure civil, electrical, structural and mechanical PEs would be quite surprised to hear there are no guardrails on their professions.

[mulr00ney]

Engineers versus "engineers".

I have fond memories of a boss who was an actual, licensed engineer while the rest of us were very much normal software devs. Boss was pretty chill except when someone someone suggested we should be called "engineers" rather than "developers", at which point they said "if you guys were building bridges, people would be dead." (I don't think all software needs to be built to rigorous engineering standards but man... I think about that line a lot.)

[rectang]

I prefer the term "software developer" and that's what I use when I don't need the prestige of the term "software engineer". It's disadvantageous for organizations to do that with actual job titles, though.

Absent US government intervention to codify the term "engineer", probably the only way out of the "engineer" trap is through further title inflation, where the developers all become "vice presidents". :)

[mulr00ney]

Yeah, it's 100% the better term. We've got rules against using engineer here in Canada though several companies I've worked for have called me an engineer. Apparently Professional Engineers Ontario sometimes goes after people for calling themselves engineers but I've never heard of it actually happening, and I don't know that they have any real teeth given that the places I worked that called me an engineer were Canadian-owned. (In fact, the only place where they checked if I could use the title was the one multi-national. Go figure.)

[jimbokun]

People meeting the definition of Software Engineer while having that title may be rare, but we certainly need more of them.

[bombcar]

In my experience “engineers” and builders are often quite “conservative” and really don’t like pushing the envelope, and they often only do it under protest.

The most famous example may be the perpetual war between architects and engineers/builders.

[fuzzfactor]

Researchers need to go wild and sometimes far off-the-rails to increase the odds of coming up with something that is both new, and potentially popular enough, if they want the option to attract marketers who can only thrive on mass-consumption.

With luck, one out of 100 inventions will show promise on those points.

There's always a lifetime wake where the overwhelming vast majority of the work remains undeployed no matter what. The more undeployed milestones and inventions that some scientists have under their belt, the more accomplished they often are whether anybody knows it or not.

OTOH, equally active engineers more often need to have most of their time engaged in actual deployment of some kind or another, otherwise not as much progress will be able to reach as many people that could benefit. So many times nothing would be accomplished without a long-term focused engineering effort once an objective has been identified. But it can be hard to stop a train when it's already coming off the drawing board at full steam.

It does seem a lot more likely for a judicious researcher to cast off some major progress in what could very well turn out to be an unsavory development, such as likely misuse, even if it could be marketed as the most popular thing they have so far. Just add it to the pile of other things that best remain undeployed. There's plenty more where that came from, and the best is yet to come.

Perhaps popularity alone is not always the best measure of progress.

[throw4847285]

I actually originally wrote "technologists" but thought that the word sounded kind of odd. Now I realize it better captures what I was trying to convey.

[Otterly99]

Honestly, scientists too. I did 10 years of research after deciding that it was not worth it. There is a vast amount of research with no direct application that gets published under the assumption that more knowledge is always better, but in my experience scientists rarely question the usefulness of their research (because most of the time, they find it interesting, which is motivating enough).

Granted, I am talking about harmless subjects, but there is also the dimension of resource usage that almost no scientist considers (the amount of plastics and chemicals used in biochemistry and cell biology is... concerning).

[tom2026hn]

I worry that companies like Anthropic, which are moving toward RSI, may prioritize speed over the timely identification of irreversible risks.

[sdenton4]

Every time I prioritize speed over risks, I, too, end up with repetitive strain injuries.

[dbalatero]

Did you mean AGI? I'd think whatever else they are doing, LLMs are reducing RSI...

[moogly]

Maybe. I am not convinced I type fewer keystrokes being a Markdown Monkey compared to writing code largely via autocomplete. Fewer Tab keystrokes, for sure.

[dbalatero]

I was mostly making a joke.

[FarmerPotato]

Yup. "I wonder how long the codebot will be combobulating over that prompt?" I'll just rest my hands for a sec..."

[ascorbic]

Recursive self-improvement

[jimbokun]

We need to force the executives and engineers working at Anthropic and other AI companies to read this encyclical and pass an exam demonstrating that they read it.

[bem94]

I have only read a few passages (and some of the excellent quotes others have shared here), but I find the underlying message here so much more compelling than those found in the various "manifestos" which come out of Silicon Valley.

I think reading this helps me imagine a version of the future I'd actually like to live in. A version where technology is used well (rather than preaching for abstinence from technology) and where values other than "intelligence" (in whatever guise) are on an equal footing.

Even writing that makes me feel naive (and to an extent I know it is) but I think it would be inconsistent for someone who cheers for humanity's efforts to solve/chip away at "impossible" problems (like LLMs were thought to be not so long ago) to shirk from the challenge of making the world better for _everyone_.

[bluesign]

The thing is why that this feels so good future is; it is a system with no constraints. A bit like Star Trek universe in Roddenberry's imagination. This kind of utopia can only be achieved with all honest actors, but in reality systems are usually designed around bad actors.

Even with all morally good actors locally, there is no guarantees for external forces. Thinking it hypothetically, even with global coordination ( all good actors ) there is not a proven path that would lead us to better place from any starting point from past.

[energy123]

It's probably more predictive to model actors as being neither good nor bad but constrained by various collective dilemmas, such as prisoners dilemma, the security spiral, tragedy of the commons, race dynamics, collective action or first mover problems, information asymmetries, the commitment problem, among others. Those are the hardest problems to solve because they're pathologies that result from the global, largely amoral structure rather than consequences of the individual exercise of morality.

In the AI case, each firm is in an arms race, and nobody can slow down without effectively collapsing due to positive gross margins only being viable with a frontier model that attracts marginal demand. An appeal to morality might have an impact but more effective action would be to address the structure that the AI companies are situated in that causes this dynamic in the first place. In practice, thats going to be a global agreement to slow down, and global regulations.

[bluesign]

Yeah but this is a system problem; if we had this utopic system from the beginning we would not even have AI probably.

[datsci_est_2015]

This sort of rationalization of evil is a core of technocratic support for Trumpism, I find, and has parallels to the evangelical prosperity gospel. Choice tenets:

  - Fuck you, got mine
  - If I don’t do it, someone else will
  - Might makes right
  - Greed is good

It’s always cloaked in a veil of realism, but it’s just the classic 14-year-old-boy-just-got-introduced-to-the-prisoners-dilemma situation. There’s nothing philosophically interesting about it.

Ironically, these are often the same people denouncing multiculturalism, yet the culture they strive for is completely morally bankrupt.

[estearum]

And it's funny because the "realism" has been proven wrong over and over and over again for millennia. People do all sorts of selfless and generous things all the time! The entire premise is trivially disprovable by just going and asking a neighbor for some help with something.

That's not to say we should be naive about greed or malice existing or being powerful motivators (especially the former), but it is obviously not true that they're the only forces at play and therefore you are "just doing the logical thing" by succumbing to them. It's just the more destructive version of the same naiveté.

[datsci_est_2015]

Seems you and I have together struck a nerve. Maybe our sentiments would have been better received in an alternate subthread, but it’s all I could think about while reading the parent / cousin comments.

I haven’t read the full Magnifica Humanitas yet, but I would be pleasantly surprised if he touched on not just dehumanization of the other, but dehumanization of the self. Expanding on your thought, succumbing to those forces under the guise of just doing the logical thing is in a way self-dehumanization - to believe you are only capable of the “logical” thing instead of the moral thing.

[bluesign]

This was not my point at all. Maybe I could explained better, but main criticism I have is: you can bundle together objectives ( which are inherently good ) and create an utopia. But those cannot always be achievable.

Everything in life in trade-offs. Simple example is speed/quality/cost. I can tell easily:

- services should be cheap - services should be fast - services should be high quality

Now I created an utopia. Obviously this is amazing to listener. They agree. But is it achievable?

It is not saying greed is good or might makes right. But system means you need to construct from this ideals best outcome ( which comes at some trade offs)

[hammock]

Your idea and its (perhaps unsatisfying to you) resolution can be summed up easily by John Quincy Adams’ quote:

“Duty is ours. Results are God's”

[bluesign]

Yeah but my point is sometimes suboptimal parts can give better overall results. ( Considering also bad actors )

[datsci_est_2015]

Yeah there were probably more appropriate subthreads to have responded to. My point wasn’t quite neatly directed against yours.

[mapt]

To get from here to Roddenberry's communism, according to Roddenberry's lore, we passed through the Eugenics Wars, the Second Civil War, and then fifty years of World War Three and the 'post-atomic horror' before coming to our senses.

[snikeris]

> I think reading this helps me imagine a version of the future I'd actually like to live in. A version where technology is used well (rather than preaching for abstinence from technology)

I believe the Amish figured this out over a century ago.

[aleph_minus_one]

> I believe the Amish figured this out over a century ago.

The Amish rather came to a different conclusion (which I don't want to judge on, but on which I nevertheless have a different opinion than the Amish).

[46493168]

What is that conclusion which differs from the post you replied to? The Amish are mindful about their technology adoption.

[aleph_minus_one]

> The Amish are mindful about their technology adoption.

The central idea concerning the Amish's relationship to technology is that only technology is allowed if it does not destroy their community.

My personal values are much less based on upholding a community, but rather are much more rooted in individual freedom and independence. This means that I (likely) come to very different conclusions regarding this class of problems than the Amish do:

For example, I am less opposed to various kinds of technology that Amish would likely consider as as "community-destroying".

On the other hand, I guess I am much more opposed to technology that can be used to surveil the user and/or makes the user dependent on the whims of big tech companies than I guess the Amish are (i.e. the Amish would likely consider this as a much smaller problem concerning which technology to allow vs disallow; as I wrote: by my understanding their central concern is which consequences some technology has for keeping their community together).

To give evidence for the previous point: (by my impression - I am not US-American) you will rather not find many Amish people at political rallys against surveillance laws. The people who attend such rallys typically also have strong opinions on which technology to use or not to use (just talk to such people who are very strongly opinionated :-) ), but - as I pointed out - these technology choices come from very different basic premises than those of the Amish.

[snikeris]

Yeah they probably wouldn't show up to a political rally because of this:

> Separation from Evil

> The community of Christians shall have no association with those who remain in disobedience and a spirit of rebellion against God. There can be no fellowship with the wickedness of this earthly world; therefore there can be no participation in the organizations, works, church services, meetings or civil affairs of those who live in contradiction to the commands of God (this may include Catholics and Protestants as well as other religions and pagans). All evil must be put away, including using weapons of force such as the sword and armor.

https://en.wikipedia.org/wiki/Schleitheim_Confession

[exceptione]

  > much more compelling than those found in the various "manifestos" which come out of Silicon Valley.

Whenever I hear these "tech overlords", I am always baffled at the total lack of culture, the absence of taste, the empty visions and the implied complete subjugation of humans to ideals of "efficiency" or "quick and easy". Maybe they would have been more interesting people if they had been brought up in beautiful towns and cities, if they had lived in a rich cultural environment instead of being raised as consumer of cheap and flashy pop culture. Maybe we should tax bad architecture, it gives me headaches but others might incur heavier damage.

As an aside, at least Trump is drawn to the grandeur of high culture from historical times, but he also doesn't understand a jota about aesthetics, and so the White House gets turned into a tacky gypsy-style abomination with one dollar ornaments.

[bombcar]

We lost the “liberal education” (not the political one, but the “freeing” classical one) and it’s starting to show.

When you compare the robber barons to Google and Meta it’s kind of embarrassing- they build massive empires of iron horses screaming across the world and covered cities in magnificent buildings (stations, libraries, etc). G&M built an empire of advertising and … not much else?

[exceptione]

Indeed. The current crop doesn't have an idea for what they hoard their billions, it's just...emptiness. I propose we explain the tech's attachment to Accelerationism as a profound boredom and lack of purpose. "What does it mean to be human"--they don't value that question. Peter Thiel got interviewed a month or two ago, and he could not be brought to say that he sees value in preserving humanity. He would rather turn himself into a robotic contraption to extend his life.

When power fears death, some strange things happens.

EDIT: link to the interview with Thiel <https://xcancel.com/rcbregman/status/2036113528126394834#m>

[bombcar]

I’m reminded (and apropos as the Pope quoted him) of Tolkien’s description of the “eternal life” the Ring gives to mortals, and how it’s … not so desirable in the end.

[jjtwixman]

Indeed It's far more necessary that the utter dregs of humanity (e.g. Peter Thiel) eventually die of old age. Or put another way the damage of mortality killing good people is more than offset by the good of it killing the worst people with the most power. Because in the end it's probably not going to be your sweet mother who will get to live forever, it'll be people like Peter Thiel. No thanks, for the good of our species.

[gmllama]

This is why I like the term "Dragon Sickness." There's seemingly only innate compulsion and no real human thought behind the hoarding. It becomes its own end. I cynically lament that it's human nature for billionaires to exist but if that is true, couldn't they at least be more entertaining about it? Bezos and Musk could be bleeding each other dry to get to the next star system by now.

[muttonette]

Google makes phones and phones are somewhat good. Better search had some value for humanity. Meta has no redeeming qualities or achievements, other than helping Trump get into office and defeat Iran.

[aerodexis]

People become tech-overloads because they are blind to these sorts of beauties - and that'd be fine if it wasn't for the fact that we have collectively allowed these people to come to power and have fallen for their empty promises of freedom and liberation.

[Xmd5a]

> Therefore builders "bear a particular ethical and spiritual responsibility"

> This is a message we need right now.

Feels good man. The solution found by the private parties driving technological change is sainthood. Or aiming for it. At least, better than you. They have the vision of what's good for the herd, but the more time I spend as a sheep, the more it seems the "herd" is just a way to recycle the story of their own exceptionalism stripped of any mark of individuality. A simple visit to fiftyyears.com will greet you with "We back the indispensable". I guess it's the same "we".

[doug_durham]

Yes but... No one living today knows what direction AI technology will take humanity. If we have an algorithm breakthrough then we may avoid building new data centers. If the abilities of the technology plateau then there might not be large impacts on employment. Builders need to focus on the impact of their next steps. Don't put polluting natural gas generators in neighborhoods today. Don't make unemploying folks the goal of your tool, today. Don't make decisions that harm people and the environment today.

[lelandfe]

https://www.theguardian.com/us-news/2026/may/13/utah-approve...

> The Stratos artificial intelligence datacenter footprint will cover more than 40,000 acres (62 sq miles) over three sites in Box Elder county in north-western Utah. The facility will require about 9GW of power, which is more than the entire state of Utah currently consumes

Sure hope your rosy inflection point happens real soon.

Otherwise, the direction this is taking us is pretty obvious.

[FabHK]

9 GW is massive. About half a percent of world electricity consumption, half of what stupid Bitcoin burns.

[moron4hire]

As I keep telling my project owner at work, "the AI will get better" is not a plan.

[GoToRO]

hard disagree. No matter what gets built, it will be used for bad things if the society is not properly constructed. We need to fix our government and then you can build anything you want. I don't like putting the blame on the builders, building is already difficult as it is. Also you can't built something in isolation, word will get out anyway. That's why this speech is either naive or he just wanted to tick a todo list: say something about AI.

[jamesinmn]

> I don’t like putting blame on the builders

Why does it have to be either/or?

From a high-level view, there are powerful cultural/political forces that nudge us towards building harmful, wasteful things without us being aware of said forces. The government, corporate greed, and billion-dollar marketing budgets are to blame.

But on the ground, from an in-the-trenches perspective, I think an engineer at Meta, for example, shares in some level of blame, too. Mainly because there is plenty of evidence showing how much harm Meta products have brought upon society. Yet Meta couldn’t be built without engineers opting in to work there in exchange for $300k salaries.

We desperately need moral clarity and courage at both the policy-level and the individual level.

[Levitz]

It is simply not feasible.

It's not either/or in strict terms, it absolutely is either/or in practical terms. When the conscientious engineer chooses not to take the 300k job, the next one in line does. If enough choose not to, it just became a 400k job.

Can you change society in such a way that nobody would take such jobs? In strict terms, sure, in practical terms it probably entails enormous costs, both economical and societal. And then you still have other countries.

There's an entire legal code filled with things on which we can't rely on the morals of the people. We can't stop theft, rape and murder, what makes you think that stopping engineers is any more feasible?

The best thing builders can do is use their knowledge and authority to pressure the other side.

[bluesign]

Because there will always be bad actors. You cannot design a system on all actors' full cooperation.

Ofc they share the blame, but it is not solving the problem.

You can say for example: loan sharks are bad for society; so government gives anyone 0 percent credit. You just removed one problem, created another.

Just and sustaining system with individual morality is destined to fail. Only option is social regulation. Which is at government level.

[xandrius]

I get where you are coming from but this is the common "reduction to politics" that anyone who doesn't want to address a problem uses: think of any societal or human problem and you can have your comment with different nouns.

Sure, IF we could just go and fix our governments in some magical way then the problem would disappear. That goes from hunger, climate change, videogame addiction and AI. The problem is that what you value in life in different than what others do, so we now have a system in which sometimes you get what you want and sometimes you don't.

But back to the topic, I do think that how OpenAI and Anthropic handled the government and them asking to drop guardrails is something a company can actually and actively do without having to reinvent the universe.

[bluefirebrand]

> I don't like putting the blame on the builders

Like it or not if you knowingly build stuff that is used for evil purposes you are complicit

You can't build an orphan mulching machine and get away with just a shrug. "I don't really agree with mulching orphans but someone else paid me to build that. If I didn't build it someone else would have" just does not absolve you of your involvement in mulching orphans

[GoToRO]

In reality things are much more mundane. You build a custom blog framework, it can be used for good or bad. You build a better app framework, it can be used for good or bad. There's nothing you can build that will not be used by bad people. Canned food and radar came from war. Depending on which side you were, it was used to do good or to do bad. Technology is just that, technology. People are still under the marketing spell of US companies that invited engineers to "change the world". People assumed in a good way. In reality you just change the world, as the other people see fit.

[bluefirebrand]

All I'm saying is you don't get a pass if you helped design a missile guidance system that your government winds up using to bomb children

Or if you're involved in building out mass surveillance that male policemen use to stalk their exes

Yeah, technology is just technology but most of us aren't just working on pure tech for techs sake. We're working on products that have a purpose.

[jjk166]

> We need to fix our government and then you can build anything you want.

Are we all going to stop building things until the government is fixed and all political problems are resolved? No? Then we must think about how we should build in a world with imperfect government.

[peanut-walrus]

It's a feedback loop between governance, social structures and individuals. But out of those, only individuals are the ones with free will who are able to "break" the loop and direct society along a different path. It's not just building "big things" that changes us, it's every small individual decision about what you choose to spend your labor on. No revolution would succeed without people willing to rebel and no dictator could dictate without people willing to follow them.

[jobu1342]

Hard disagree to your disagree. Every organization is made up of people and reflects the character of its members. So with the nation itself. If people won't take responsibility for their own actions, neither will the government. I.e. self-government isn't the answer to fixing a broken people. Transformation begins within one's self, not in imposing one's will on another. First everyone much look to themselves, then the government, organizations, and projects will fix itself.

Everyone likes to talk about "fixing the government". It feels nice to understand the problem with something else that is broken. The problem is that replacing the people in charge is a no-op if you don't have a pool of good people to choose from and the will to choose them.

[Npovview]

"That which can be destroyed by the truth should be." - P.C. Hodgell, from her novel Seeker's Mask (1994)

"That which can be automated by the AI should be."

[moron4hire]

When he speaks of the builders needing to take responsibility, I think he is directing his comments at tech company leadership. The Sam Altmans of the world, not the poor slobs like us who need a job for health insurance because apparently unemployed people don't deserve to live.

[salawat]

No, he's talking to you too. The Sam Altman's can't be a threat without useful idiots to cooperate with and doing work for them. You don't get out of your own culpability because you locked on a set of golden handcuffs.

[jimbokun]

Oh, just “fix government”?

Glad there’s a simple, easy to implement solution to solve this problem! One that hasn’t been considered or tried before!

[pheaded_while9]

Consciousness is the causal factor, not government. You are putting the cart before the horse to insist on any political solution.

[hideout_berlin]

why not government

[dogcomplex]

True. World governments committing genocide and building concentration camps yet it's the computer scientist who's supposed to worry about ethics here because they're the ones who are merely the first to inevitably mix two paper's methodologies together?

All of this is empty puffery til the US and Israel are condemned. Go after the Big Tech billionaires backing those monsters, sure, but no builder needs to be more concerned about ethics than the very institutions designed to concentrate human decision making. Fix your own house first, folks. Techies - keep building, and do better than these people.

[dubcanada]

I frankly disagree, how do you know "will this make humanity better?" until the product is done? AI wasn't what it is today because of a specific companies innovations. It stemmed from decades of research built on top of other research. How did any of those builders know what they were getting into?

Every technology should be done to the fullest potential, how are we ever supposed to explore the stars or cure cancer if everyone is scared of accidentally building Skynet?

[alok-g]

Just like we can incrementally innovate, we should be able to incrementally figure too.

[ncr100]

Cool, inspires me to think, "should we regulate away the toxic side effects which we realize COULD have come, if we built it less considerately?"

[slashdave]

> We cannot condone naïve enthusiasms, nor fuel unfounded fears

[ngcazz]

Like glibly synthesizing a reimplementation of NextJS

[oersted]

I'd argue that the problem is exactly that the responsibility has been left to the builders, who have a hard enough job actually building and have little room for competing incentives.

The concept of a self-regulating industry is utter nonsense. I am a builder, and I try to do things right, but if I am honest with myself I do not have the mental capacity or willpower spare, or the worldly knowledge of everyone's perspectives, to manage these massive challenges. I need other smart people to also focus on that, people that are actual representatives of the population, and yes, force me to stop and do it better occasionally.

Achieving a good balance sometimes does require having opposing forces fight it out, it can be healthy.

[hackboyfly]

Thanks for this!

[kombine]

"Technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it."

My comment under the post on Omarchy here got downvoted and flagged by the "technology is neutral" crowd because I dared to say that it is unethical to use software produced by a white supremacist.

[pickledish]

This is also the message I got from "the wind rises"! Though from talking with other people that takeaway doesn't seem universal -- which IMO is one of the ways to tell it's a great film :)

[pj_mukh]

Under this logic, would the Pope have built a Hammer? a Knife?

I am hoping the Pope has a cleaner view of AI quietly automating the drudgery in the back offices and not just robot dogs with machine guns. And with that view, should we build it?

[PantaloonFlames]

It's too simplistic to imagine the tension is between robot patrol dogs vs automating drudgery. If automating drudgery suddenly puts 30% of people out of work, it has huge broad negative impact on people who are currently alive and working in the current system. Innovate, but do it with awareness.

[pj_mukh]

Than an even better question for the pope is:

If a technology existed that reduced the cost of producing a critical thing (think food, housing, medical care) down to near zero, however, it made the humans currently building the thing redundant, should we build it? Would it be okay to use the hyper-optimization power of Capitalism to build such a technology faster?

[3form]

Likely the answer would be: yes, we should build it, and take appropriate care of the people being made redundant.

These two are not mutually exclusive, it's just that no one wants to pay for it.

[zplo68]

Science and Engineering is morally detached.

Ask a scientist or engineer what philosophy or theology has taught us about the source of morality and their education, training and experiences havent prepared them to answer that question.

This didnt matter so much in the past because their activities never had the scale it does today. For basic training in philosophy if you are mid or upper level exec whose decisions are going to effect a whole lot of people, go to open yale courses and take the intro to Philosophy classes. It will help develop your answer to the question - why do you do what you do if you are going to die anyway tomorrow.

[tensor]

This is a ridiculous comment. Science papers always have sections on impact. It's the running with scissors industry types simply chasing the bigger paycheck that don't stop to think.

[slfnflctd]

Far too many of us (particularly younger people, but not only them) undervalue or are dismissive of philosophy. I once was like this, partly because at the time I'd been brought up - like most humans - to believe my parents' religion held all the answers philosophy might address.

I quickly learned as an adult that whether you're a person of faith or not, it's not pointless at all. It's the foundation of everything. Philosophy is how you explain the deeper reasons behind why you follow whatever religion you do, or adhere more meaningfully to whatever kind of agnosticism, atheism and/or 'spirituality' (with or without woo) you espouse.

[moron4hire]

This can only be true if you believe in a certain mad-scientist version of these things. We as a society have systems (however flawed) to hold scientists and engineers to ethical standards: e.g. peer review and in many cases--e.g. civil engineering, architecture, medicine--even legal frameworks to enforce ethical standards. Science and engineering are human endeavors, they are not divorced from the human condition, they cannot be separated from humanity and human rights.

Maybe you mean mathematics is amoral and are committing the common conflation of "engineering is just applied science and science is just applied mathematics," which is a really bad case of missing the forest for the trees.

[lstodd]

It's true not for belief but for the fact that "we" do not have "a society".

Humankind evolved to learn the universe because that was the only way to survive, and no secrets of it will be left unknowable since leaving them that way is a threat to survival.