[Aurornis]

So one of their servers had a /heapdump endpoint that publicly served a heap dump of the server? This whole saga is out of control.

This group didn’t really “publish” anything, though. They’re offering access to journalists through a request form. They’re also not saying how much actual message content they have because the 410GB of heap dumps makes for a bigger headline number.

[mingus88]

Can you imagine co-opting a trusted and secure (and free) bit of software and just making it worse at seemingly every turn?

And charging for it?!

I’m not sure what is more embarrassing: to be the company or to be a user.

[miki123211]

This is why Signal is so opposed to third-party apps (or forks) that connect to their service.

If you want to keep the branding of Signal being the secure app, you need to make sure that all Signal users are actually using a secure version of Signal.

If an insecure fork (like this one) becomes too popular, most groups will have at least one member using it, and then the security is gone.

[pchristensen]

That was Apple's same reasoning for shutting down that iMessage client app. These leaks seem to justify their concerns.

[xandrius]

Nah, that was to keep their users hostage and force them to buy a iPhone.

[fn-mote]

This is a shallow dismissal of an argument that should be given more consideration.

Sure, this is HN, we know one of the effects of locking the ecosystem and coloring in-system messages differently is to encourage people to be in the ecosystem.

At the same time, you ALSO need to consider that obviously there will be leaks.

Malicious/advertising apps will target the new messaging interface to gain more data on their victims, etc.

[smaudet]

Safe encrypted group chat with stangers is an oxymoron.

Locking down a platform is not an acceptable solution to the above conundrum - it doesn't matter if the user is using an official device/app whatever if they are untrusted. They can always turn around and leak everything you say without any technical measures.

Should we have no security? No, if you want to color messages differently based on perceived platform, fine. This is just an illustration that no technical measures can replace the fundamental trust necessary in these types of situations.

[aesh2Xa1]

Hm, my understanding is that TeleMessage archival works with iMessage in the same way it does with Signal.

The third-party federation problem is real, but the vulnerability caused by TeleMessage isn't solved by removing federation.

[xorcist]

If your product is a strong brand then that would make total sense.

I believe the main criticism against Signal is that they should focus on getting widespread traction of secure messaging, and that perhaps the brand can be a relatively distant concern.

[calvinmorrison]

That doesn't seem to be a problem for protocols and having a single implementation can lead to bugs that defy spec yet cause no issues obviously.

[ctxc]

But you're not branding or selling implementations

[ctxc]

*protocols

[hypeatei]

Why would the company be embarrassed? The users (i.e. high level U.S. officials) did no due diligence. Of course a private company is going to take the easiest and cheapest route. If it goes bad, just shut down and spin up a new entity.

Some speculate this was intentional intelligence gathering by the Israelis which is plausible too.

[n2d4]

> Some speculate this was intentional intelligence gathering by the Israelis which is plausible too.

How does this make sense? If they were gathering data, why would they add a public download? Surely the Israeli officials would not want foreign powers to access this?

Per Hanlon's razor, I don't think this is attributable to anything other than incompetence.

[barbazoo]

Two things can be true at once. Them using their access to unencrypted messages for nefarious purposes and them being incompetent at the same time leaving that endpoint open.

[jojohohanon]

There’s room for both sides of the razor. The heapdumpz could be there maliciously, but incompetently made globally accessible.

[pigbearpig]

From the Wired article: "The archive server is programmed in Java and is built using Spring Boot, an open source framework for creating Java applications. Spring Boot includes a set of features called Actuator that helps developers monitor and debug their applications. One of these features is the heap dump endpoint,"

So the heapdumps being available is a Spring Boot feature so it does not appear to be malicious.

[flarecoder]

I'm the original author of the Spring Boot feature for heapdumps: https://github.com/spring-projects/spring-boot/pull/5670.

It seems that users commonly misconfigure Spring Boot security or ignore it completely. To improve the situation, I made this PR: https://github.com/spring-projects/spring-boot/pull/45624.

When the PR was created in 2016, endpoints were marked as "sensitive" and, for example, the heapdump endpoint would have to be explicitly enabled. However, Spring Boot has evolved over the years, and only the "shutdown" endpoint was made "restricted" in the later solutions. My recent PR will address that weakness in Spring Boot when users misconfigure or ignore security for a Spring Boot app so that heapdumps won't get exposed by default.

[evrflx]

This feature must be explicitly enabled, it is not on by default nor by accident.

[g-b-r]

I mean, it could theoretically have been to provide plausible deniability, but it seems extremely more likely to have been incompetence and carelessness (and if they were also sending everything to Israel, it was probably through some unencrypted ftp upload).

[michaelt]

Imagine you ran a spy agency and you were infiltrating signal, Facebook, Google, aws, cloudflare, and so on.

Would you have them make a secure back door that could only be intentionally designed, and potentially traced back to you?

Or would you just have them be incompetent in plausible, deniable ways?

Nobody’s getting shot for espionage because they chose log4j and it had the shell shock bug.

[notpushkin]

I mean, one doesn’t preclude the other. This could be an incompetent intentional intelligence gathering.

[aucisson_masque]

The Israeli would have made it secure so only them can access the data because knowing someone else's secret is worth something only when it's still a secret, if china, Russia and everyone can read the log of the American government it's worth nothing.

[dylan604]

>Some speculate this was intentional intelligence gathering by the Israelis which is plausible too.

Which does not bode well for the customers' counter intelligence abilities

[donnachangstein]

> The users (i.e. high level U.S. officials) did no due diligence.

But why would they? It's not their job. They have massive IT staff supporting them. "High level U.S. officials" are just executives; the pointy-haired bosses to the pointy-haired boss. Only difference is these wear little decorative pins over their breast pocket.

Every Fortune 500 company has dedicated IT staff for execs; someone you can call 24/7 and say "my shit's broke" and they respond "we just overnighted you a new phone".

These people couldn't even install an app on their MDM-controlled device, now the narrative has become we expect them to be making low-level IT decisions too?

Next week we'll be scrutinizing Pete Hegseth's lack of thoughts on rotating backup tapes.

[Jedd]

> ... narrative has become we expect them to be making low-level IT decisions too?

I think that's a misdirection.

The narrative is that:

a) they were using a compromised piece of software

b) they should not have been using that software - not (necessarily) because it was compromised, but because it wasn't US DoD accredited for that use case.

(I understand your point that these guys are not tech savvy, and do not need to be, but they should be regulation-savvy (clearly they either are not, or willingly broke those regulations), and they should be following organisational guidelines that presumably cover the selection and use of these tools types.)

[da_chicken]

Yeah, and the purchase approval process is in place specifically so that someone who knows what to look for has looked at it and verified that it's an acceptable configuration.

This is the exact same problem as Clinton's blackberry enterprise server. Doing it right was hard and time consuming, so they ignored that and did what they wanted.

Only we should be a lot more demanding that our officials in 2025 have a better basic understanding of the importance of computer security than in 2005.

[nkrisc]

> now the narrative has become we expect them to be making low-level IT decisions too?

If their staff makes bad decisions, that’s their failure too.

We expect them to be ultimately responsible for what happens on their watch.

Was it Truman who said, “Woah, don’t bring the buck anywhere near me, it stops with my assistant”.

[microtonal]

It is too early to tell, but given that these people openly attack scientists and other experts (they don’t agree with), I wouldn’t be surprised if they ignored advise of their IT experts.

[input_sh]

It's not too early to tell, we knew from the beginning that the use of Signal (let alone its clone) was not authorised to be used for such communications.

Yes, there's a fleet of people who are supposed to make such tech decisions. The people involved specifically went against those rules. The existence of a group chat using an authorised app is a violation on its own, adding a journalist to it is a violation on top of a violation.

Adding a journalist was accidental, but using such an app (despite it not being approved) is very intentional.

[cornholio]

IT staff that knew it was illegal to provide them tools for a conspiracy were fired or silenced. So the only people left were their cronies, who instantly complied with their illegal request, to the best of the cronies' abilities. For such national failures, the buck has to stop at the very top, not on some IT monkey.

This is typical for highly corrupt governments and autocracies, they crumble from within because the autocrats can't trust random, competent people so their inner circle becomes saturated with people who are selected on the basis of loyalty not competence, and these people end up making the most important decisions and running the country.

[3rdDeviation]

Would tend to agree with most of that, but I think the assertion is Petey needed to ask his IT leadership to do the due diligence before diving in, not that he needed to decide using his own depth of skills and experience.

I assume he did and they said it was a bad idea - the memo they'd released a few weeks prior about Signal vulnerabilities seems to suggest a lack of faith in that approach - but he was already banging away on his phone with all the grocery reminders and definitely not battle plans he needs to keep pushing out. Which is also how it feels in the enterprise space these days.

Strange thing to see our bureaucracy start to behave like a corporation instead of the other way around.

[hristov]

Their massive it staff provides them with a way to communicate securely and they ignore it deliberately so that their communications are not preserved for history or for future court cases.

[TeMPOraL]

One man's low Integrity (in the "CIA triad" sense) of communications is another man's improved plausible deniability.

[kube-system]

The changes to the application are intentional by all parties because message archiving was required by law.

[brookst]

Sure, but they were not required to be done incompetently and insecurely.

[sneak]

The fundamental concept of plaintext archiving (escrow) of messages from e2ee messaging apps is insecure by most definitions.

They could have used user-custody public key cryptography, where the end devices have the pubkey of the customer, and archive only re-encrypted messages to TM that they can’t read.

That is not, of course, what they did. They just archive them in plaintext.

[kevincox]

I don't think it is. I can archive my own messages and E2E security on the messaging layer means I don't have to trust the operator of the messaging service to not read my messages because they can't. The choice of how I archive the messages is completely orthogonal to the choice of messaging platform security. I could choose to use an E2EE approach if I want but in that case it probably wasn't even desired as the point was to have these be archived for audit purposes. (Of course they are more secure options such as archiving to an audit key, but this is still orthogonal to the concern of the messaging protocol)

[_kb]

Well, I suppose technically this /heapdump endpoint does satisfy that archive requirement.

[yapyap]

User for sure

[HenryBemis]

(read with sarcastic tone) But hey, this is a 'lite' version or a 'red' version (icon is red) or a 'purple' version (icon is purple), so I am cooler that then others that have the standard.

I haven't used WhatsApp for 'a very long time' as I have exited the FB ecosystem, but back in the day I remember seeing "lite" or "WhatsApp+" or other variations of the software. I wouldn't be surprised that those "lite" or "+" come with baggage.

[BearOso]

> They’re also not saying how much actual message content they have because the 410GB of heap dumps makes for a bigger headline number.

That's very important to say. I went through one of these massive data dumps recently and it was literally all cached operating system package updates and routine logs. Nothing at all of interest.

It's easy to cut the size on a heap dump. When it's not done it seems sketchy. But it could be a 512GB dump and already pruned, so I could be wrong.

[harrall]

Most of the the heap dump will be filled with stuff like java.util.String!blahjava.util.ArrayList!

Though the heap dump would have messages in flight at the time. It's obviously not as useful if you are just trying to grab messages for a specific person.

Frankly the most useful part might be any in-memory secret keys, which could be useful for breaking deeper into the system.

[aorloff]

Plenty of info from a live heap dump if you know what you are doing.

But these guys are only interested in "journalists" not people who spent decades digging into ad server heap dumps

[barbazoo]

Aren’t those Israeli software companies all supposed to be top notch, ex Mossad, yadda yadda? Doesn’t sound like it.

I hope the message dump is juicy.

[msy]

And SBF of FTX fame was ex-Jane St so obviously was a serious finance professional. This is why using past employers as a shorthand for capability is unwise.

[sillystu04]

In fairness, FTX had a profitable bankruptcy [1]. So it's still better to be scammed by Jane Street alumni than to be scammed by the usual alumni of Goldman Sachs, JP Morgan etc

[1] https://www.bloomberg.com/news/articles/2024-05-15/ftx-bankr...

[stackskipton]

It's not profitable. They are getting their money back from value of the assets in 2022 when they went bankrupt but most of crypto assets have gone up significantly in value so it's 2.5 years of lost profit.

[coolcase]

How is that fair? It was luck from the AI investment. Pure luck.

[fredoliveira]

Regardless of how you feel about SBF and FTX, claiming an early investment into Anthropic is "luck" rather than being ahead of the curve feels off the mark.

[coolcase]

That is dodging the point. The guy ripped people off. By luck they got the fiat value of their investment at some past date back. Yes if a single investment pays off well enough to negate fraud losses on that scale over a short time scale. It's fucking luck.

[arresin]

It wasn’t the only smart investment

[gruez]

I thought Israel has mandatory military service, so ex-mossad or ex-military signals intelligence doesn't really say much? Presumably they're directing people based on their skill set, so you'd expect most hackers to end up in mossad for their mandatory service.

[kennywinker]

> Presumably they're directing people based on their skill set

Big presumption.

If I were israeli, there’s no way in hell anybody with half a brain would want me near their spy agency.

When a gov is committing a genocide, their decisions are based on control and fear, not getting the best out of people.

Edit: downvote all you want. Israel is still committing a genocide. No hospitals left standing. Killing aid workers, journalists, and doctors. A million people on the brink of starvation. Literally salting the earth to prevent crops from being grown. That is war crimes, ghettoization, and genocide.

[viraptor]

That's not a great generalisation for the whole country. How many ex Mossad people interested in doing actual implementation in tech companies do you think there are? It's like "aren't those US software companies all supposed to be top notch, ex NSA yadda yadda?"

[conradev]

They do start a lot of tech companies specifically: https://en.wikipedia.org/wiki/Unit_8200#Companies_founded_by...

The US only has voluntary military service, so the dynamics are different

[lysp]

The CEO/Founder of TeleMessage Guy Levit was the head of the Planning and Development Department of an elite technical unit in the Intelligence Corps of the IDF according to bio.

[aorloff]

I guess we could say that in many ways, he never left

[oceanplexian]

One problem that smart people tend to make is in thinking that being really smart in one area is generalizable to all others. Just because they're good at AppSec doesn't mean they're good at networking or operating a webserver.

[ripley12]

I agree with this. It's surprising how often I encounter people with that belief, because I was disabused of it very early on in my career; this industry is chockablock with people who are brilliant in 1 area and deficient in others.

[coolcase]

That's why you need teams. Red team for example! Security team. App developers. Code reviews. You need all the process too. Security that relies on one genius is fragile.

[czl]

Aka "halo effect"

[karn97]

That sounds more like a stupid person than smart lol

[stefs]

you can be smart in one area and stupid in others. the "not knowing you're stupid in others" is part of the "stupid in others".

[rsynnott]

I'm not sure why you'd expect intelligence agency types to be particularly good at engineering, tbh.

[rainworld]

Spooks in general like to project a veneer of competence, downright invincibility. Entertainment media, journalists, experts play a big role in this. And by and large it works.

It’s especially true for spooks of a certain entity. Also, it’s easy to confuse brazenness, being protected from consequences, and usually downplayed or secret Western complicity with competence.

[rsynnott]

I mean, I'm sure they're competent in some stuff, but being competent in one field doesn't generally mean being magically competent in _all_ fields.

[keeda]

I'm not sure about this case, but maybe the assumption here is that these are people from a technical branch of Mossad, such as Unit 8200, which does SIGINT. I've interviewed 3 of them for your typical Big Tech SWE position, and to a candidate, they were very strong engineers. I never got to work with them, however, because they always got better counteroffers...

[ExoticPearTree]

> Aren’t those Israeli software companies all supposed to be top notch, ex Mossad, yadda yadda?

Working with a few companies like these, I can tell you that the marketing is top-notch, and very aggressive. The products not so. Most get better with time.

[underdeserver]

"All supposed to be".

This is a country of 10 million people, a rather heterogeneous one at that. There are going to be better and worse companies.

[H8crilA]

They are top notch - at working for profit and for the interests of their country.

[treebeard901]

After all the concern over China and TikTok, why is the USG using a foreign chat program at all?

[coolcase]

SuperPAC and other corruption

[coolcase]

Yeah the /leakitbaby endpoint was meant for just them, not the world! Doh!

[elzbardico]

It only takes one guy doing one stupid thing to have a security incident. Yeah, processes should be in place, but no process is perfect.

[jfim]

Sounds like someone had a Java app and mistakenly exposed all of the JMX endpoints over HTTP. It's not the default configuration, and likely done out of carelessness.

[pigbearpig]

From the Wired article, it may not have even been a mistake, depending on the version of Spring Boot.

"Spring Boot Actuator. “Up until version 1.5 (released in 2017), the /heapdump endpoint was configured as publicly exposed and accessible without authentication by default."

[davedx]

This sounds utterly insane. Is Actuator a standard part of Spring Boot or is it an optional package of some kind?

[teekert]

Imaging putting up a firewall to mitigate this, then docker compose helpfully opening the ports for you. Security comes in layers.

[callamdelaney]

This feature of docker compose is insane.

[teekert]

Right!? I learned with a colleague: Didn’t you restrict everything to the Tailnet? Yes, feel free to check UFW. Hmm, then why does nmap show all this stuff when scanning from the lan? Wtf??

[callamdelaney]

Similar here, UFW setup to only enable access via Caddy to our http services - wait, why can I connect directly to our redis instance?

Took a while to workout that for some reason docker-compose is messing directly with iptables to shoot holes in the firewall we'd configured. Figured out you have to write your compose in some super special way to disable that functionality. Compose should never ever open network ports, ever in my book - to do so without a warning or anything though is like I said, insane!

[formerly_proven]

This was also part of the exploit chain in the "Volksdaten" incident.

[0xbadcafebee]

Or intentionally. There could be an APM agent which just lets you run heap dumps any time you want, or they enabled heap-dump-on-crash, or had a heap dump shutdown hook, etc. There's a lot of ways to trigger dumps. If we're talking about a full dump, and the apps were using most of the memory allocated to their container/VM/etc, 410GB is actually not that many dumps (we're probably talking uncompressed). At 4GB/dump, that's around 100, over possibly several years.

I just wonder where they were storing them all? At one place I worked, we jiggered up an auto shutdown dump that then automatically copied the compressed dump to an S3 bucket (it was an ephemeral container with no persistent storage). Wonder if they got in through excessive cloud storage policies and this was just the easiest way to exfiltrate data without full access to a DB.

[trebligdivad]

Is this a heapdump of servers or of clients? I can imagine that might have been intended as a place for crashing clients to log

[kleton]

TeleMessage is most likely an intelligence asset, and a burned one now that Trump's people stopped using it. A fake hack is the safest way for the agency responsible to leak the messages collected.

[aorloff]

and provide a plausible reason for the shutdown

[kbouck]

if a heap dump is a copy of all the bytes in memory, then wouldn't "thousands of heap dumps" likely be larger than 410GB?

napkin math:

  410GB/1000 dumps = 410MB per dump?

  410GB/2000 dumps = 205MB per dump

[diggan]

Might be filtered somewhat, like extracted all ASCII text then compile that into the dump, rather than just the raw dump files.

Edit: reading the description on the dump again, seems exactly what they did:

> Some of the archived data includes plaintext messages while other portions only include metadata, including sender and recipient information, timestamps, and group names. To facilitate research, Distributed Denial of Secrets has extracted the text from the original heap dumps.

https://ddosecrets.com/article/telemessage

[coolcase]

Kubernetes pods?