[solarkraft]

This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ...

> Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you.

This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem (this is why I’m building a phone app for OpenCode).

This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.

[benob]

People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277, in addition to facilitating distillation (you don't pay the full cost of RL)

[rsfern]

Or they could store the reading traces and validate the user hasn’t edited them server-side? They could sign reasoning traces so they can’t be counterfeited?

[tough]

both openai and anthropic actually do this and let you as a harness store only the encrypted contents of thinking traces to be passed again on further turns but they only have the key to unencrypt it so its jibberish for final users

https://developers.openai.com/cookbook/examples/responses_ap...

[ses1984]

Why would the model bother to check signatures if it doesn’t check tags?

[embedding-shape]

The latter ("facilitating distillation") is a real concern from the labs I'm sure, but the first part I don't understand what you mean, or you misunderstand that paper, it's about "prompt injection" via manipulating the reasoning, not about the model reasoning by itself and somehow that leading to more prompt injections. They're quite literally maliciously rewriting the reasoning as the model reasons, not just showing it to the end-user, two very different things.

[spijdar]

Yeah, signing would prevent the paper's attack, but seeing the "raw reasoning" is helpful in figuring out what triggers refusals or (especially) what in the system prompt is guiding a specific refusal.

Gemini (the web UI) used to show raw reasoning, or at least a more detailed summary of its reasoning, less than a year ago. Complete with markdown and weird spelling idiosyncracies, so I'd lead towards "real reasoning", but who knows. The reasoning block leaked the system prompt way more often than the response block did, and you could figure out why it would refuse a request through the reasoning, even if the response itself refused to elaborate. This is, presumably, why they stopped showing it. No loss for them, just prevents "pesky users" from low hanging fruit snooping.

(Gemma 4's reasoning and output remind me strongly of what I remember Gemini 2.5/3's reasoning to be, as an aside. I guess that's obvious, but Gemma 3 felt like a totally different model, while 4 feels very Gemini-ish.)

[benob]

What matters for this injection strategy to work is to follow quite closely the style of the reasoning. It's particularly effective if you copy reasoning from the same context. If you cannot see the reasoning, you cannot duplicate it's style.

That said, including instances of the attack in training is already a good countermeasure.

[jurgenburgen]

Token-based reasoning also seems like it would be inefficient, there’s no reason it has to be English or even human understandable.

[skinfaxi]

What use is the reasoning if its unintelligible to a human?

[jurgenburgen]

Reasoning improves the output of the model. Originally it was a prompting method called chain of thought but later models were trained to do the reasoning steps without the human prompting for it. Being able to see the reasoning steps is just an accidental benefit.

https://arxiv.org/abs/2201.11903

[miki123211]

Even if it is intelligible, reasoning styles (and hence reasoning effectiveness) differ between models.

For example, gpt-oss loves reasoning in the style of "I be caveman, hungry, need food, need coconut, will search coconut now, eat when find." Giving that to a model unaccustomed to that style could cause it to respond like that.

Reasoning interpretability helps debug why models fail (and some labs will give it to you if they trust you not to distill or be hacked), but there are also conflicting goals like token-efficiency, so interpretable reasoning doesn't always mean "pretty sentences".

[UltraSane]

There have been models trained for latent space reasoning. It has a lot of advantages but the big drawback is the reasoning is completely opaque.

[placebo]

>Like a frog enjoying a nice warm bath...

Exactly :-)

Call me naive but I think dark patterns are a short term strategy for winning and I'm optimistic that in the long run they will be replaced with those that are more respectful and oriented to the greater good (granted, the long run might take more time than one hopes for).

Given that the pendulum can sometime swing back fast enough to be able leverage it, it might be a good time to focus on models that are both open weights and economically viable to run to build the next great thing.

[blauditore]

> [...] dark patterns are a short term strategy for winning and I'm optimistic that in the long run they will be replaced with those that are more respectful and oriented to the greater good

This is definitely not the way most software has been going in the past decades. It's rather the opposite: Companies play friendly to obtain a user base, then start applying more and more dark patters to further increase their profit. Facebook, Evernote, Instagram, Komoot... The list goes on and on.

[pcthrowaway]

Also, see Android and iOS which have a monopoly on the mobile OS market

[falcor84]

Sorry about the nitpicking, but s/monopoly/duopoly/

[utopiah]

I don't think that's the economical definition of monopoly. It's not about 1 single actor owning the market but rather about unilateral change. Obviously a single actor owning 99.99% will be able to shape the market ... but also one owning a lot less, e.g. 30% if even if they don't collude with another actor owning e.g. 21%.

[monknomo]

what you are describing sounds like an oligopoly not a monopoly. A company cannot realize monopolistic benefits without > 50% control of a market. That's not to say there aren't benefits to a large market share, but they are different and we have different terms for them.

[chrisweekly]

not nitpicking; huge difference between 0 and 1 competitor

[jmathai]

Huge difference while both are competing. But it’s a huge risk in a market with such high cost of entering. If either divests then it rapidly degrades.

[cyclopeanutopia]

"Naive" is a big understatement.

[solarkraft]

I can’t think of any example in which it has gone like this, but some in which it has: The most prominent in my mind would be software transitioning from “of course you get the code so you can extend it yourself” to what we have today (please correct me if I’m wrong about this, I’ve only read about it).

Users at large (individuals and companies alike) don’t care much about a freedom taken away when a product is a few % better than another, so the biggest player sees if they can get away with it. And once they do it, everyone else does too.

[wren6991]

As a frog I'm happy to report the water is comfortably warm today, the same as it was yesterday.

[embedding-shape]

> This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.

Same, I dislike it so much I've acquired 96GB of VRAM to run local models, but sadly nothing so far, even with that amount of VRAM, comes even close to running Codex with GPT models. I really, really, really want local models to be ready, and things like Laguna S2.1 NVFP4 gets really close of almost being there, when it comes to coding specifically. But still feels like we have a long way to go for local models to be serious general purpose alternatives.

[solarkraft]

The open models are good! Even when hosted, they buy you a lot of portability.

[rkuska]

I am waiting for prices to get down and in the meanwhile I am gathering as much session data from claude/codex as possible so I can then later use them to fine tune open models. I've built my own session parser/archiver for this.

[sdoering]

Same. And I built a search tool for that repository of older sessions, so that I can search through those on all my machines. This helps me to be able to pick something up on a machine I did not originally start the convo on.

[gb2d_hn]

It was with the transience of valuable agent session dialogue in mind that I built https://www.agentkanban.io - A key feature is context capture in the tasks which you create on the board and then progress in agent sessions. Supported agents are currently Claude and Github CoPilot in VS Code. All of your context is captured in the task and can be reloaded into new agent sessions at any point in the future. Tool use is intentionally discarded because it is proprietary in nature and therefore breaks the promise of session portability.

[quietbritishjim]

This analogy doesn't make sense to me. Once I've had a conversation, it may be useful to refer back to it shortly afterwards, but typically the output is some code (or maybe a recipe or instructions for something or whatever). It's certainly important to me that I can use that code elsewhere. That's the thing that, to me, is analogous to being able to switch operating system and still access my files or use my favourite word processing program.

But being able to decode the blobs that subagents returned in a conversation I've already forgotten about? That doesn't affect my ability to switch at all! I can't even imagine what trivial detail of an operating system this is analogous to. Maybe an undo buffer of a document that gets cleared on exit anyway?

I can still pass all my code, and any associated documentation, to any other agent at any time. What am I missing?

[theshrike79]

https://indieweb.org/POSSE

This is the way.

[AtlasBarfed]

At this point you should realize there are now tech executives that have 30 to 35 years of experience in all the dark patterns.

Anything new is going to be an extremely rapid race to the bottom.

[tcfhgj]

You are building a phone app for open code?

You mean like fossify phone or?

[NamlchakKhandro]

no he means he's locked him self into a bloated and slow react based terminal ui.

waste of time. should be using pi mono

[solarkraft]

No, it means that I’m building a fast alternative to the stock solid based web UI. Pi (isn’t the “mono” just about it being a monorepo?) has no features and no defined server-client API.

[agilek]

And yet people still go with Sign in with Google as the only login method.

[msdz]

I’ve actually started ignoring services where that’s the only login possibility.

Sorry even if your platform is the greatest thing ever, but I’ll find a different tool. I’ve read one too many stories about Google (or Apple!) closing the entire account over some bullshit unnecessary reason like “fraudulent” gift card issues or whatever. I’m certain the affected people would’ve preferred to just pay back the amount in question instead of losing their entire Google Drive, or their 20 years of iCloud Photos or whatever.

[n6242]

It's the only reason I want to replace Tailscale with something else or look into self-host when have a bit of time during my vacation. They only allow login through a third party, which is a big no for me.

[corney91]

Tbf to Tailscale they allow any OIDC provider[1]. I wish this was more normalised, then we could have one login everywhere regardless of who hosts it (even if it's yourself).

[1] https://tailscale.com/docs/integrations/identity/custom-oidc

[dolmen]

While that isn't convenient is you don't want to use the public identity providers, it should make you think about what makes your identity on the Internet, and consider to have your own identity provider on a DNS domain you control.

[noduerme]

This is what headscale does, right? Manage private logins and keys for your tailnet from a vps? I haven't played with it but seems straightforward

[zalebz]

I created my own AWS Cognito userpool just for tailscale. I recall the webfinger redirect was frustrating to get right but I refused to use Google/Apple as a gatekeeper to my own network so I had all the motivation to get it working.

[aidos]

I think you can use a passkey now.

[stavros]

Only for people invited to the Tailnet. To create your own account, you still need a third party.

[eddythompson80]

You can host your own identity provider. Authentik, Keycloak, authelia or Zitadel work.

I don’t fully agree with tailscale’s decision to not want to be an identity provider but I understand it on some level. It simplifies their service greatly and the amount of asks for that will essentially have them build a full enterprise Entra-like solution that would be a constant maintain headache and they are not interested in that.

[dd8601fn]

It’s odd, I just saw my first one of these today. Some former Figma person linked an AI site comp designer.

I was going to give it a shot, but the only choice was a Google login.

So very weird.

[mike_hearn]

It's not that weird. Building a modern accounts system is a lot of work and people don't like creating new accounts. So it's the path of least resistance for developers and users.

[noduerme]

It's not even that it's hard to build a modern account system. It's that if you put your site behind a private / custom account system, and you post a link to it on HN (for example), everyone and their mother complains that it's requiring an email address and password. But if you put the same site behind a Google auth, most people wouldn't think twice to click the button.

[mike_hearn]

I worked on the Google account system for a few years. I'd say it's a lot of work to beat it. Not necessarily "hard" for a team with the right skills, but certainly a lot of sweat, blood and tears.

A modern account system is expected to have, in rough implementation order: email confirmations, password strength checks, password reset emails, forgot password flows (=advanced ID verification as otherwise this becomes a backdoor into accounts), user profiles (+avatar image upload/recompression/hosting), usernames independent of email addresses along with ability to change usernames later, password brute forcing blockers, bulk signup prevention (=solid bot detection), abuse controls (can easily become a team of people), 2FA (SMS), 2FA (authenticator apps), 2FA (backup codes), 2FA (voice calls), 2FA (passkeys), 2FA: recovery when both factors are lost, enterprise SSO integration (SAML), enterprise SSO (Active Directory), fast global signout support (much harder than it looks), cookie theft mitigations, heuristic online login risk analysis to catch cases where an attacker knows the right password via phishing, support for signing the user in to multiple domains, audit logging so users can review their own sign-in history, age verification and restriction support, and possibly support for being logged in to multiple accounts in a single browser session.

Oh, that all has to be HA, and the account system is the keys to the kingdom so the security requirements are the strictest of any part of your system.

You might say we don't need all of that, but expectations rise over time. Maybe 20 years ago you could get away with a simple account system and an automatic forgot password flow that just assumes the user still has access to their email. Maybe today you still can write a simple system, if you don't expect to have many users and are willing to implicitly delegate identity to webmail providers anyway (the moment you assume the user has access to a secure email account you're basically doing Sign In With Google anyway for 90% of users). But if you roll your own accounts, and then the user gets phished and someone logs in from an obviously suspicious place with the right password, they won't say "yes that's my fault" anymore, they'll say "Google could block that log in, why didn't you?" or maybe "Why didn't you support 2FA? It's your fault".

[skinfaxi]

> A modern account system is expected to have, in rough implementation order: email confirmations, password strength checks, password reset emails, forgot password flows (=advanced ID verification as otherwise this becomes a backdoor into accounts), user profiles (+avatar image upload/recompression/hosting), usernames independent of email addresses along with ability to change usernames later, password brute forcing blockers, bulk signup prevention (=solid bot detection), abuse controls (can easily become a team of people), 2FA (SMS), 2FA (authenticator apps), 2FA (backup codes), 2FA (voice calls), 2FA (passkeys), 2FA: recovery when both factors are lost, enterprise SSO integration (SAML), enterprise SSO (Active Directory), fast global signout support (much harder than it looks), cookie theft mitigations, heuristic online login risk analysis to catch cases where an attacker knows the right password via phishing, support for signing the user in to multiple domains, audit logging so users can review their own sign-in history, age verification and restriction support, and possibly support for being logged in to multiple accounts in a single browser session.

You don't need half of that. Even to this day, anthropic lets you log in by sending a code to your email. No password, no dealing with resets, no MFA. So yeah, you definitely don't need all of that.

[StilesCrisis]

Thank you for this. The whole attitude of "it's just [complex thing we take for granted], how hard could it be?" is exhausting.

[dd8601fn]

I dunno, I’d say it’s never been easier.

If you’re building a paid service you’re probably already using cognito or supabase or something. That puts you a few clicks aways from 5+ other identity providers and normal accounts.

[wwind123]

I am guessing that's because Sign in with Google is the least of the evil, as compared to other popular OAuth Sign-in options like Facebook, X...

[ulrikrasmussen]

A password manager has an almost as efficient UX

[stavros]

It's the least evil of the evil options? Why not use one of the good options then?

[pcthrowaway]

Google is more evil than X or Meta by some assessments

[eddythompson80]

It’s funny how the tides shift. I remember the more dominant online position in the early 2010s being “I don’t want to create yet-another-account for your website. Just let me sign in with my google account”. I’m with you though

[the_mitsuhiko]

I guess the good aspect of sign in with Google is that it falls back nicely to sign-in via email which is very portable.

[qurren]

Because with the alternative they'll require you to "confirm your e-mail" instead of just picking a password

[eru]

I'm actually more annoyed by having to pick passwords than confirming my email.

[mystifyingpoi]

I'm opposite. If my email goes down, or I do not have access to it, I cannot login. With passwords I still can.

[eru]

My email is with Google. They are less likely to go down than my password manager.

[Viliam1234]

Until one day your Google account gets locked with an automatically generated message that you violated an unspecified rule. Good luck getting contact to a human support.

[mystifyingpoi]

Well then you need a better password manager. Like a .kdbx file which is ~100% available, given enough copies I guess.

[jurgenburgen]

Then what’s the point of it all? Just use Google login.

[lucumo]

Doesn't your password manager take care of that?

[eru]

It does. But I haven't properly synced it across all my devices, yet. I know, my fault.

[zalebz]

when in a rush I will put in gibberish instead going through the pw manager steps and rely on resetting the pw at a later time

[hotelsacher]

Picking passwords is a job best left to password managers. You don't use a hand crank when riding an elevator, do you?

[eru]

I am using a password manager.

[Geezus_42]

Use a password manager to autogenerate them?

[0cf8612b2e1e]

Considering the two different people who fat finger my email address when signing up for services, I wish more systems required a validation step.

Janet and Jake, I am sick of getting your stuff. Learn your actual email.

[Geezus_42]

> This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem.

I wish people would apply this same logic to governments.