[huy-nguyen]

I work at a government agency and here are my tales.

1) They install a root certificate on all machines and use that to MITM all TLS connections using a firewall appliance. They turn this MITM on one day without notifying any developer. Overnight, all our builds (run on-prem) fail because npm install, pip install etc fail and we spent a long time trying to figure it out. They are still failing to this day and I have to get off the VPN every time I need to run these simple commands. IT absolutely doesn't give a flying **** about developers.

2) They ban all non-Chrome browsers from being installed. As in, if you install such a browser and try to launch it, the system will say "browser X is banned. Contact IT." They would have banned Safari too had it not been part of the OS. Furthermore, they also disabled private browsing in Chrome (probably the ability to do this is why they allow Chrome). I think they're preventing people from hiding their internet browsing.

[sillystuff]

The MiM might not be your IT folks, but rather management. I was in a meeting which included folks from Palo Alto (PA) and management where PA was hard selling their ability to MiM all https connections and link all activities of the users to their usernames through various methods from directory integration to log scraping on radius servers. The managers were super excited about the possibilities. Management not only wanted to implement this, but wanted to do so in secret. IT folks were pushing back-- hard.

Firewall as bossware.

Firefox being banned is because it uses its own certificate store, so Firefox users would see a browser warning every time they visit any https site notifying them that their traffic is being MiM'd. Chrome and chrome reskins like MS Edge use the OS store which MS Windows centric organizations can easily (centrally using MS tools [GP]) add the trusted CA for MiM into. For the Macs, it probably wouldn't matter since the 3rd party mgmt tools could probably push out either.

[justsomehnguy]

> Firefox being banned is because it uses its own certificate store

FYI You can instruct FF to use system trust store: https://support.mozilla.org/en-US/kb/setting-certificate-aut...

[throwaway743]

Caught an ex employer using sslstrip and they definitely used bossware. Management would imply they were reading work and personal messages, emails, browsing through thinly veiled threats to workers (self included) and through gossip.

They also used push notifications on the desktops to know when people were active or what they were doing, and had keyloggers installed/active. Once caught a manager's personal laptop on the network running mitm software. A friendly coworker in IT confirmed all of this with me in private.

Tried warning a couple coworkers, but got brushed off. People don't seem to care nor believe even though they're being manipulated.

That place was a nightmare to say the least

[TheRealDunkirk]

Oh, how I have learned the hard way on this.

Our IT now blocks outbound SSH entirely. You know, the secure way to access VM's in, say, our cloud? Sigh. I'm sure there's a "jump" server somewhere that I'd have to log into, `sudo` to another account, THEN SSH to my target box. Whatever. I just avoid the VPN.

I used to use `cntlm` to tunnel requests through our firewall for things like Ruby's bundler, as it required NTLM authentication. Now they've also gone the additional mile, and installed a certificate (Cisco Umbrella) in all of our computers, and require its signature to pass the firewall. Unfortunately, it took me a long time to sort this out: why `cntlm` no longer worked, and why none of the usual suggestions on SO fixed it. I finally figured out that RubyInstaller for Windows included a nice facility to deal with this. You just place additional certs in a directory, run a Ruby script, and it will bundle the whole stack into a single .pem, which it will reference for all network-related commands. Thankfully, bundler's error messages were telling me the specific certs I needed, and I could download them from Cisco's web site.

Just about a month ago, my company started requiring that cert for ALL traffic, not just HTTP(S). Like for, say, Postgres connections on port 5432. I finally realized that I could reference that same SSL bundle in my Postgres client connections, and get through.

I've spent about 8 years here now, and it's been a cat-and-mouse game the whole time. I'm always wondering what's coming next.

[xist]

The way you brush it off is insane. Using a jumpbox IS more secure. I understand this may cause problems for your workflow (though there are many ways to work within the confines) it sounds like you're stubbornly insisting your way is the best (and therefore most secure) way. This reeks of entitlement. Work with people and stop being a prima donna, you're not above security concerns.

[rtev]

As a security engineer, most developers I work with are like this.

[slt2021]

your practices are the epitome of Shadow IT that company management doesnt like and fights

[TheRealDunkirk]

You don't know the half of it.

[dspillett]

> IT absolutely doesn't give a flying ** about developers.

They are not paid to. Their performance is judged against how close they get to zero compliance issues, not how close they get to zero times developers were unhappy!

> I think they're preventing people from hiding their internet browsing.

Without delving into the “do you have the right to privacy even on a company machine”, who would be daft enough to do something they want to hide from the company on a company machine, or the company network at all? Though there are valid useful uses of pron^H^H^H^Hicognito mode so it seems silly to ban it.

[wahnfrieden]

You’re just arguing for surveillance (by capital or state) with the tired line of you should have nothing to worry about unless you deserve it which is absurd/reactionary

[Spooky23]

You’d probably feel differently if some developer at the IRS compromised your identity because he used some compromised library to avoid internal processes.

Security, public trust, etc requires controls and audit to deliver. You don’t want a startup “fake it till you make it” mentality in government or banking.

[ornornor]

More like you can’t possibly expect your work machine to have any kind of privacy, being in the trade and knowing all the ways companies can (and will) use anything they managed to gather about you should they need to obliterate you.

It’s simply good practice not to use your work machine for anything personal at all ever. Because depending where in the world you are, anything stored or viewed on a work machine gives your employer de facto access to it, legally speaking.

[wahnfrieden]

You can be careful as a worker and still be against workplace surveillance

[ornornor]

For sure, but let’s face it… we’re getting more surveillance and bossware at work, not less unfortunately.

[dspillett]

I'm not saying it is right for people to be monitored, but that I would never trust that I wasn't being so I'd not be daft enough to do something I don't want the company to know about using their resources.

And there are perfectly valid reasons for companies to monitor traffic: data exfiltration, accidental or malicious, is a significant concern for companies that hold and process PII and for the people who have their PII held/processed by those companies. It is not as black & white as “monitoring and surveillance bad” unless you only care about your personal privacy.

[sillystuff]

Some of these "security" products that MiM TLS traffic allow configurations that objectively reduce your security. You can configure Palo Alto devices to accept a self-signed cert from the Internet, but present your trusted MiM cert to the on-site user. Now the user isn't aware that they are the victim of a second MiM outside the organization.

The organization also exposes itself to greater liability. E.g., a rogue employee could use the trusted MiM CA cert for their own MiM e.g., capturing banking credentials of co-workers or accessing user/employee PII they would otherwise not have access to.

Yes, monitoring traffic by MiM https to external sites can alert you to / possibly prevent accidental exfiltration, but it doesn't prevent intentional exfiltration. It is, however, very effective at monitoring employees. The thing it is best at, might be its true purpose in an organization.

[dspillett]

> but it doesn't prevent intentional exfiltration

It can prevent accidental exfiltration, or deliberate exfiltration by a relative incompetent, which are the majority of such problems.

You are right in that they will not stop deliberate actions by a competent disgruntled or a competent external attacker who has access (but you have a much wider set of problems in this latter case).

Maybe I'm old-fashioned (I am definitely a “working in an office, living at home” person which seems to mark me out as a dinosaur in the coming remote-work age!) but I don't think it is my employer's responsibility to provide me with unfettered unfiltered internet access to do personal stuff with. Work stuff on employer provided Internet which they can monitor all they like, personal stuff on my own devices & connections which they can keep the hell out of.

[Xelynega]

Does this mean that it's OK for an employer to put cameras in employee bathrooms? The argument can be made that it's not the employers responsibility to provide me with unfettered access to a space to do personal stuff with, just like internet access, so why not?

[sillystuff]

If the employer needs to back-door encryption to discover your personal activity, how is it their business what you are doing? Your activities obviously caused them no public issues. If they did, the encryption back-door would not have been necessary for the discovery.

In more civilized areas of the world privacy rights are explicit, and even things like employers snooping on employee email accounts on company owned email servers is illegal. When at work, you are selling your time to your employer, but that doesn't imply that the employer owns you while you are at work.

As to the sibling comment about cameras in workplace bathrooms, yes employers did this and now there are laws prohibiting it. Now, employers just account all your time using bossware leading to e.g., folks at Amazon having to pee in bottles or wear diapers on the job to not get fired. There is no line that some capitalist employer will not cross unless we place limits with consequences to reign them in-- e.g., we no longer have employers forcing small children to crawl into running machine tools to clear a jam while risking a limb being sucked into the mechanism and turned into hamburger meat-- but, we did, it was common-- lives of the poor (especially children) were cheap, but stopping an assembly line was expensive.

[slt2021]

I am afraid you are talking about stuff you have absolutely no idea about.

Palo Alto appliance should be configured with both Forward Trust and Forward Untrust CA certificates, and the issue you described will not exist. If some people misconfigure - thats their fault for not following instructions.

Secondly, rogue employee doesnt have access to CA key that is stored in Palo Alto appliance. Only your firewall admin will have it, but if your main firewall admin went rogue, capturing colleagues’s data is the least of your concerns. Insider threat of that calibre is equally applicable to rogue CEO or CFO stealing all money from the bank. Or your ActiveDirectory admin getting CFO’s credentials and corporate bank credentials.

[sillystuff]

You seem to acknowledge that you currently can configure the device in the manner described while simultaneously being extremely aggressive. A conversation I had with PA support gives me the impression that PA didn't have 'Forward Untrust' when they first started back-dooring TLS i.e., the PA support person did not counter my point of negative security implications of their MiM back-door for invalid certificates encountered externally. This conversation was something of an on-site debate between PA reps and a few of our tech staff. PA pushing for spying on users and tech staff trying to come up with technical reasons why it was a bad idea (management already loved the idea of spying on the users, so no appeal to decency was going to work. Management arranged the debate without telling staff it would happen until the last minute while it was planned ahead with PA for weeks; IT staff at that college were good people who had a history of advocating for user privacy).

Having PA MiM TLS connections is the organization back-dooring itself as well as the external sites the users connect to. This back-door is available for abuse by IT staff, management and/or an attacker(internal or external).

There is a rule that seems to eventually always be proven-- if you provide infrastructure that can enable abuse, eventually it will be abused. Even if you and everyone else involved in the decision at your organization have good intentions, your future coworkers / management may not. Presumably the FBI and NSA have more thorough back ground checks of their employees than the average employer, and both have had employees abuse their access to surveillance data to e.g., stalk ex-girlfriends. And, even if the employee isn't rogue themselves, when the order comes from above, many will obey immoral/illegal orders-- e.g., Ronald Reagan, as president, had the FBI spy on his daughter's boyfriend. The safest option is to not to install the back-door in the first place.

PA's ability to tie Internet activity to specific users' identities was central to their sales pitch-- our tech staff assumed this was targeted at windows shops, but we used non-MS stuff including our LDAP servers and hoped this could kill the surveillance project-- PA countered that they could, at a last resort, do things like e.g., scrape radius logs to associate identities.

PA appears to be a competently run company that probably knows what messages are most effective at selling their product, and they really pushed user surveillance. Therefore, I suspect that many organizations who purchased PA products based that decision on the user surveillance capabilities (explicitly to enable abuse by management).

PAs main feature seems analogous to an illegal phone wire tap, and IMO should be illegal (especially without notification to the victims-- both on-site and off-site). It is curious how corporate circumvention of encrypted communications without permission of the external site hasn't been seen as a CFAA violation while a simple 'view source' on a browser can result in SWAT pointing a gun at your child.

[thaeli]

Why are they allowing you to run npm, pip, etc from public repositories at all? That's a huge supply chain risk. If builds are worth doing on prem they also need to be pulling solely from internal, vetted repositories.

[dspillett]

> Why are they allowing you to

Maybe “you shouldn't be doing that anyway” is a key part of why they don't care to spend effort resolving the problem.

[SSLy]

Bold of you to assume the developers were given a budget and staff to run on-prem repo mirror.

[TheRealDunkirk]

The idea that some team is "vetting" that the entire stack of stuff you'd pull from npm for a React front-end app is "safe" is ridiculous. Forget the mirroring; that's trivial. What criteria or process would make you think you had a "vetted" snapshot, beyond what they already do!?

[slt2021]

This process is automated by static code analysis tools, once it is deployed then absolutely no meatbag effort is required

[flumpcakes]

This is a big worry of mine.

I've worked in various places from big finance, public sector, to privately owned. It was only the big finance institution (the biggest) that actually seemed to care about supply chain attacks. Everything was locked down super well and you could not! use a random library without it getting vetted by a central team. In fact, we were even locked down to specific versions of programming languages.

People see this as annoying and in the way of developers, but it really is the only way to secure your development "supply chain". When people cry about this I always ask: do you really want the entire financial industry grinding to a halt because someone took down left-pad?

[Spooky23]

For good reason. Stuff like that is a really high risk and won’t meet audit standards.

I’m in charge of the IT goons somewhere. We aspire to provide a better level of service and maintain local repos of things you’re allowed to use. Stuff like Node isn’t allowed near anything important though.

I would be careful. An agency doing stuff like that is probably running an EDR that will detect and report on what you’re doing. If it catches what you’re up to, you’ll be jammed up.

[_fat_santa]

It might very well be for a good reason. But in my experience, it's never the policies but the communication. IT was right to make whatever policy change they needed to, the fucked up by not telling any of the dev teams.

[Spooky23]

100% agree. Most fubar things are caused by poor communication, lack of empathy and poor understanding.

[jjoonathan]

The #1 value proposition of the cloud is escaping dogshit expense processes and the #2 value proposition of the cloud is escaping dogshit IT.

[deepsun]

Well, if we're talking about security, their ban on NPM is a good thing, that's a huge supply chain risk.

If you don't have a budget for the vetted repositories, it means you don't have a budget for the project within the security requirements. You shouldn't be circumventing the security requirements, you should escalate the issue.

PS: of course I'm not talking about other things like MITM certs, that only reduces security.

[raxxorraxor]

> They ban all non-Chrome browsers from being installed.

There are portable apps for other browsers, Firefox for example. FF has its own certificate store that overwhelms IT.

about:config -> security.enterprise_roots.enabled and it uses the system store.

Overall Firefox is a very good browser to configure for different machines. The out of the box Chrome or Edge are just a bit more forgiving to MITM attacks. So the user doesn't notice perhaps? Aside from that they are horrible browsers with horrible priorities.

[ornornor]

Worked at an insane place that did 1.

“It’s more secure” they said.

The “solution”? Disable certificate checking of course! What could go wrong?

Same place that ran a vulnerable instance of nexus (the package manager) for all internal npm and maven packages for a whole year before patching. It was publicly accessible. And it had a banner on the homepage that said “this version is vulnerable (severity 10/10 anonymous RCE), update NOW”. Anyone who went to https://nexus.1337company.com would see it.

That company did software for the government. I’m sure I wasn’t the only one who noticed the vulnerability and that some packages got tainted. But we’ll never know because no audit was ever performed and there were no backups of that server anyway.

Like I said, absolute joke of a workplace.

[bombcar]

That first one indicates something is being injected and the checksums are failing, that's ... worrying.

Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess.

[marcosdumay]

It could be just the problem of the certificate being invalid for those tools because the MITM one was installed only for Chrome up to the firewall replacing all of the files with html pages of some antivirus with internal links where the user can download them.

Corporate middleboxes come in all shades of stupid.

[thayne]

> Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess.

Combined with the fact that chrome is the only allowed browser, I suspect it is the other way around. Chrome uses its own certificate stack, and I would guess IT only added the MITM certificate to the chrome trusted CA list, not the system one.

[Karunamon]

I would be willing to bet the first one is caused more by those tools not being aware of the firewall appliance CA rather than failing checksums. Doing your own certificates at scale is a pain in the ass because every tool/container has its own way of handling the trusted list.

[slt2021]

this is because "pip install" is insecure, this is supply chain risk. Your IT team should have provided local artifactory proxy through which you can pip install.

you should use this command "pip install -i http://artifactory.mycompany.local pandas" and get url for artifactory from admins

[TheRealDunkirk]

If "your IT team" has merely created a snapshot of an external repo, how is this any more "secure?" I've asked a similar question below. I really want to understand the thinking here. No IT department is going to go line-by-line through all the packages in "artificatory" or Ruby gems or NPM packages or NuGet's repo, checking them all against known vulnerabilities. No one's going to vet the actual code. If there's a public advisory for one of the packages, the parent repo is going to fix it first, and the internal repo may already be compromised, and the "IT team" is going to have to duplicate the work that the repo runners are already doing, and do it slower. I'm lost here.

[slt2021]

there are IT security vendors that provide static code analysis and scanning for known signatures, that can detect and block malicious packages. Just target SCA at local artifactory and this will be a solved problem. CISO just needs to buy solution and IT admins just needs to deploy that software once and it will keep scanning. Absolutely no extra work from meatbags is required

[richbell]

> Absolutely no extra work from meatbags is require

Unfortunately this is rarely true in practice. There is always some degree of friction or error that ought to be managed; ignoring it is how shadow IT proliferates, e.g. a dev is tired of their builds failing due to a false positive and decides to circumvent artifactory altogether.

You're spot-on otherwise.

[TheRealDunkirk]

So these bulk scanners exist, and the issue is a solved problem, but none of the "root" repos for the popular language stacks are using them?

It seems that Microsoft has built an internal tool that runs such a scan on NuGet (https://devblogs.microsoft.com/nuget/how-to-scan-nuget-packa...), at least against your individual app's packages. (That would be a very rare h/t to Microsoft from me.)

EDIT: Apparently, you can also do this with npm packages (https://docs.npmjs.com/auditing-package-dependencies-for-sec...). I don't see any facility to do this with Ruby gems.

It looks like the common practice would be to outsource the issue database to GitHub, and let whatever scanner you're using cross-reference that list?

What happens when it finds a reported problem? Does it automatically delete that mirrored package, and/or block it from being downloaded or used from the on-prem repo?

This is all new to me, and has helped put this in context, but what actual software are you talking about using for analysis?

EDIT EDIT: Running `yarn audit` in my main Rails app (just using webpacker to bundle the JS):

    97 vulnerabilities found - Packages audited: 1074
    Severity: 2 Low | 34 Moderate | 52 High | 9 Critical

I just did a `yarn upgrade` about a week ago, so it's not like I'm completely out of date. What would a centrally-managed SCA do about this situation?

[richbell]

> So these bulk scanners exist, and the issue is a solved problem, but none of the "root" repos for the popular language stacks are using them?

I could talk at length about this; unfortunately, I'm on my phone with a shotty connection.

The tl;dr is that companies like Snyk make money by requiring companies to pay to check for vulnerabilities once they've been downloaded. There's not necessarily anything wrong with that, but a philanthropic company could make things significantly safer for everyone if they weren't concerned about making money. Initiatives like the OSSF will hopefully have a positive impact, for this reason.

[slt2021]

You need money to staff people with security knowledge that constantly keeps product up to date with latest malicious signatures in all codebases.

This is why only commercial company can build a great product. The constant cat and mouse game between threat actors and defenders/researchers. Every new malware/trojan/cryptominer strain needs to be found, identified, signature written, and all clients need tk get latest signatures asap, and product has to work flawlessly with as few false positives as possible

[xani_]

And that kids is how it looks when security team just sits in their ivory tower and shits on everyone else in name of security theathre they're paid to play

> Overnight, all our builds (run on-prem) fail because npm install, pip install etc fail and we spent a long time trying to figure it out. They are still failing to this day and I have to get off the VPN every time I need to run these simple commands. IT absolutely doesn't give a flying ** about developers.

Add their cert to system store ? Won't help inside containers tho... without much fuckery

[slt2021]

You are not supposed to mess with certificates inside containers.

Security team should have provided you with golden image with hardened config, latest patches installed, and corporate certs installed in certificate store.

If they didnt, they aint doing correct DevSecOps/SecDevOps or whatever the fancy term is for integrating security within development team.

It is a big red flag that any developer can pull whatever image for container running in production, possibly with unpatched vulnerabilities and loose config and ports open, and running with root privileges, etc.

Usually stuff has to be vetted and checked prior to being deployed in production environment

[richbell]

> Add their cert to system store?

That's the fun part: every technology and tool has its own bespoke way of handling certificates, and it often isn't as simple as adding a certificate to the system store.

[bornfreddy]

Why not? Just mount a volume with certs and you're good to go. In every container, of course.