Hacker News new | ask | show | jobs
IT’s Dirty Little Secret: “We’re aware of ‘Shadow IT’, we just can’t stop it” (openera.com)
28 points by KThornton 4928 days ago
14 comments

Its fun to rail on internal IT. Most organizations inadvertently set the department up to fail and then find themselves shocked, shocked I tell you, to find that they have failed to deliver.

The boys in the basement aren't a bunch of Luddites, before the upstairs staff has even heard of the new tech out there, they're already dependent on it in their personal life (or have demoed and tossed it to the curb).

Spoilers: They actually can stop it, they're the ones managing firewall config after all. You should ask yourself "Why haven't they?" Probably has something to do with the fact the buisness requirements and/or budget preventing them from using the tools everybody would prefer.

I'm not anywhere near the firewall team, but my company seems to struggle with blocking these services quite a bit. In particular, google drive is very hard to block: they use HTTPS and they don't have a fixed set of addresses that the service is delivered from.

Their solution? Add a GPO to all our windows machines to force a '127.0.0.1 drive.google.com' entry into all the HOSTS files on our network.

Hadn't really considered things from this angle - I'd be happy to see a win-win-win for the people, IT, and the company.
If you want to solve the problem you're discussing in this article, you seriously need to talk to some dejected netadmins.

Most corporate technology problems where a solution exists but isn't used aren't technology problems at all, they're office politics problems (for the sake of argument I also consider business requirements / SOPs to be under the office politics umbrella, if you've ever tried to change them you know this is true).

It's rare that more technology actually fixes the problem. Usually getting more/new technology is a catalyst to changing the underlying social problems, or is just a workaround.

For example, my alma mater wants to implement a new thing to make service better on campus (sorry about the vague-ness, its about privacy of the people involved, and I'm not even supposed to know this). If the project goes through as originally planned, they'll save money and greatly improve services. But, it will never be approved without letting the CIO win a turf war in the process, so the project will end up spending an extra >$500k on unnecessary tech to do it her way. Did I mention this is a public school that really can't afford to be paying that much just to feed egos?

Of course the "security" to which CIOs refer is not DLP or anything cool like that (I'm not implying that DLP works, only that it's cool) but rather their own job security. IT is a cost center, and CIOs only survive when they can account their costs to other parts of the business. If e.g. marketing, sales, and accounting can honestly say they don't need anything that IT is providing, IT might not be around much longer.

From an actual security standpoint, it makes sense to really evaluate how secret your data need to be, and then set up an infrastructure to support that. Individual customer demographic data should be absolutely secret, but that doesn't just mean that marketing people shouldn't upload it to Dropbox so it's easier to pull into their abominable Access DB. That means that the only people who ever see it are CSRs while they're actually talking to the customer. Then IT can add value by isolating CSR desktops on their own 802.1X-secured wired network, while providing a more open network for their other work, and encouraging a shred-all-post-it-notes policy.

I think IT can make legitimate security arguments, but these can't start with "gosh Dropbox is terrible!" Dropbox and other cloud services are used because they are useful. Rather than depriving the individual employee of useful services, find services the business as a whole needs but doesn't realize it needs.

> While nearly two-thirds of companies (60 percent)report they have corporate policies in place that prohibit such actions, respondents say there are no real deterrents for purchasing cloud services by stealth. In fact, 29 percent report there are no ramifications whatsoever and another 48 percent say it is little more than a warning.

If it's such a big deal that employees are using Dropbox in the office, employ some of those Orwellian tactics bigcorps are so good at: block them. Block them and their entire CDN. Shut off access to Facebook, Google Drive and Box while you're at it. Make them use only corporate e-mail. Is being denied access (at work) to a service they purchased not ramification enough?

Shall we draw and quarter them instead? You're not powerless, you're just myopic.

I'd wager that if a corporation has a problem with employees using Dropbox, they've got problems with a lot of other stuff - so why not stamp it all out at once? Or, work with it! Embrace the growing cloud culture. Buy Dropbox for Teams, or Github Enterprise, or what have you. Clearly, your employees want it.

Or, disband the thought and grow up.

EDIT: Comment below generated while the site was not responding to requests.

> 503 Service Unavailable

It appears the "Shadow IT" has won this round.

> "why not stamp it all out at once? Or, work with it!"

You can't work with it, because of liability. If you bless Dropbox and champion it to the rest of management, it becomes your problem when the inevitable data breach happens.

But you don't want to stamp it out all at once, because: 1. CIOs know that cutting off things people want really badly just leads to better circumvention tech (more people running proxies or using 3G laptops, etc) and suddenly you can't even watch what they're doing, let alone stop it.

2. Those things are useful. Just because the Enterprise can't make peace with limitations or find a suitable analogue doesn't mean those tools don't legitimately make people more productive.

Is this a situation of people not having the resources or permission to do the jobs for which they were hired?

So these people are trying to get a job done despite their manager's actions. A manager's job is to remove roadblocks to getting things done. In this situation, the manager needs to self-remove.

Policies and punishment have proven to be useless tools to stop the spread of rogue clouds. Employees will do what they need to do to get their job done.

CIO's are adopting cloud apps. The reality is that users will still inadvertently save files in the wrong place. I know I do all the time. If we can help get the files into the right place, even if the user saves them in the wrong place, then that is progress and lessens the negative impact of rogue clouds.

Make it easy for people to do the right thing and don't make them change the way they work.

Hmmm. Is this from signing up on web or iPhone app?
From an iPhone, but the problem has resolved itself. Thanks!
Great, thanks!
The idea there is something wrong with the resourceful workers instead of the lagging IT is perposterous.

IT right now in many companies is living in 2004 still. SO MUCH has changed in the intervening 8 years, it's no surprise that people are going with consumer grade products when corporate IT doesn't deliver modern resources.

> it's no surprise that people are going with consumer grade products

Indeed not. IT lags because it's hellaciously expensive to have it any other way. They're more than aware of what's happened over the last 8 years. At my day job, a profitable software shop doing some fairly cutting-edge stuff, we run everything on Lotus Notes. My desktop PC has 2GB RAM and runs Windows XP: a decade-old operating system. We just migrated our source control system from Visual SourceSafe to - wait for it - SVN. It's a gigantic leap forward!

IT recognize that they're not in a position to dictate radical, wholesale tool-and-process change. So they turn a blind eye to private initiatives which help employees stay productive, while gradually and systematically replacing broken pieces of infrastructure.

I use my own personal MacBook Pro for most of my work, relegating the XP clunker to a Notes terminal (a job at which it struggles.) I use Dropbox for syncing my own work and for sharing gigantic virtual machine images with my staff. I run three agile development teams using various cloud-based apps to manage workflow, dropping back to Lotus for necessary book-keeping tasks and ticket assignment. I run a backlog database in Evernote, and we have an internal wiki for mockups and collaborative story editing. In other words, my own personal mix of bleeding-edge and relatively mature.

That's what most businesses are like: a compromise, a heterogenous mix of solutions and processes which evolve over time. There's no shining uplands where every employee exclusively uses the latest tools, while very few workplaces are stuck with uniformly last-era tech.

Even if IT suddenly decided to spend millions of dollars in a company-wide orgy of upgrading, the resulting chaos would bring our business down quicker than the spend would ruin us.

The described IT painfully reminds me of Soviet-style planned economy. It tries to be the only economy in tow", but as it falls behind due to inefficiency, it tries hard to suppress any other economies that try to arise.

And of course it is done in the name of security! Obviously everyone is trying to steal your secrets and that's why you have to live in outdated and broken environment.

Actually, people are trying to steal business secrets, and they are doing so all the time. The Chinese are notorious for it, and they target basically any business that has a valuable trade secret, even those that are not military. Even the US has been caught using its intelligence apparatus to pass foreign trade secrets to domestic businesses:

https://en.wikipedia.org/wiki/ECHELON#Controversy

If I were running a business whose trade secrets were worth more than a few hours of some Eastern European hacker's time, I would be concerned about computer security.

Security measures prevent you from doing work. You have to find a balance. USSR never found one, but USA did. So there is no longer any USSR.
A lot of different aspects of companies remind me of this. Usually dictatorial control, rigid hierarchies, policies made with no input from those who will follow them, etc. It's wonderfully ironic that the iconic capitalist organization is often so communist internally.
> It's wonderfully ironic that the iconic capitalist organization is often so communist internally.

I'm not an advocate of communism by any means, but I think the word you're looking for is "authoritarian"; maybe "dictatorial".

I don't think so. That certainly forms a part of it, but there are also the aspects of e.g. senseless policies, large sub-organizations doing nothing useful for no good reason, people engaged in turf wars instead of doing something productive, etc. Authoritarian or dictatorial regimes can be quite efficient if the dictator is good, and I don't really associate those features with authoritarianism, but they are definitely stereotypical (if not necessarily real) communism.
The workers don't own the means of production in a firm. It's not communist; it's Soviet. Show trials; pointless dig-and-fill exercises; five year plans; Potemkin villages; lunatic dictates from unaccountable leaders; and shadow economies.

Wonderful read: http://blogs.valvesoftware.com/economics/why-valve-or-what-d...

Precisely. I'm using "communism" in the American stereotype of communism sense.
I really do hate reading articles that praise rogue employees using cloud services.

It's wrong for an infinite string of Data Loss reasons, uncontrolled access to cloud services is no different than leaving a laptop filled with confidential information lying in the front seat of your car.

It doesn't matter how secure the user thinks it is, nobody in Security or Risk Management has qualified or quantified the risk.

To say that Executives would rather stifle productivity is false, they will get the appropriate tools for the job for their workers, that has never been the issue at any organization I've worked for directly, or consulted for.

The real reason nobody cracks down on this, is kind of ironic, although the executives know it's going on, and they will chastise or have you written up for breaking policy/procedure, the truth is that they don't really know what their security posture is and they don't want to know for liability reasons.

There's a lot of willful ignorance, because Security in IT truly is a giant black hole cost center to these people, and rather than seeing it as protective measure, they see it as something that stifles productivity and costs enormous amounts of money.

Security in IT can be a way to reduce cost (via risk mitigation), but all too often it's just a form of authoritarian power play by petty tyrants.

In my experience, executives will get "dust in their eyes" if you bend a few rules to get things done in a bureaucratic environment. Plausible deniability, effectively. They want productivity without having to pay for it.

Dropbox, for example, is mostly free (up front), but with a level of risk cost associated with it. An enterprise on-premise Dropbox alternative is not free (up front) and may or may not have less risk than Dropbox. What's the better one? It's hard to measure. What's the ROI of sharing files? Depends on if your management likes fancy numbers games or just approves projects based on personal preference with numbers to make it look like they're doing some due diligence.

We're not trying to praise rogue employees for shunning corporate policies and opening up huge security holes.

The reality is that it's happening regardless. People are going to do what they feel they need to to get their job done.

Thus far, the general approach to dealing with this is to enforce more policy, block where possible, etc... which again, has done little to reduce employees from "going rogue".

We want to open the conversation on better ways to solve this problem since current methods simply aren't working.

As a network guy who gets that best understands the risks and consequences of unsecured, unsactioned clouds being used in a company - what would you suggest as potential solutions to give employees tools they need to get their job done, and the Company and IT the security it needs?

Good points. There is a lot of willful ignorance. (Plausible deniability is one of my favorite excuses I hear!) I think its a bigger problem that really depends on the type of company or organization. Small and large commercial firms face less risk than Defense, Pharma, Financial and other highly regulated industries.

I don't think anyone is praising employees who go rogue, but I for one completely understand why they do, and sympathize. In many cases companies have made it way too hard to get things done. When systems get in the way of getting $#!t done, people find a way. Especially if their livelihood (sales, consultants...) depends on it.

The "cloud" is a huge problem in the finance, legal, healthcare, and educational fields. Confidential client/patient/student data leaking out all over the place is a disaster waiting to happen, not to mention often outright illegal.

Let me give you an example: I recently bought a Livescribe Skypen, the new one with Wifi. It automatically syncs with Evernote, and works like a charm. But I can't use it for purpose, taking notes at work, because I can't have attorney work product for a client floating around on Evernote's cloud. That's just a no-go. My father in law encountered a similar problem. He's an IT director at a school district, and he has been trying to get teachers/staff to stop sending student information through GMail/Google Docs. It's almost certainly a violation of student privacy laws to expose that information to third parties without student consent.

I think there is some disruption to be had in this space. People want to use their iPads/tablets/etc and other cloud-reliant devices in their work flow, but at the same time that information has be stored in a way that adheres to security protocols and privacy policies. Google could over a "local Google Drive" service where a company could let its employees use Google Docs, but have that data stored in the company's internal network, with assurances that Google can't troll through the information to target ads or any similar privacy breaching and potentially illegal activity.

I don't think Google would be too interested in providing that service, but I don't see why someone else couldn't do it. At some level though, a Google Docs that's restricted to the office or campus is strictly less useful than old-fashioned docs on your laptop's harddrive, edited by normal GUI editors. Would any user want to use that service?

In general, I think you have start mistrusting employees more, though. If an employee can't be trusted not to attach rightfully-secret data to email without heroic IT efforts to prevent that scenario, maybe that employee can't be entrusted with the data period. The old "firewall" method of implicitly trusting everyone on staff with pretty much everything is quite inappropriate for most business situations.

It doesn't have to be restricted geographically--iDevices support VPN just fine after all.

And I think there is a disconnect between what users can be trusted to do in person, and what they can be trusted to do with computers. I don't think most users have a good mental model of how the cloud works, how it exposes data to third parties, etc. I imagine most people don't even realize that Google reads your e-mails and documents.

Just to clarify: are you more concerned about the Googlebot reading your documents to sell you consumer products than you are about employees attaching business or customer data to email or shared docs?

Because I'm operating with a much different threat model. Email is not and never has been secure. It is sent in plaintext unsecured from one unauthenticated mail server to the next. The moment the user attaches data to an email the game is over and we have lost. Sensitive data must be kept in systems that are designed to store sensitive data, and which do not have a "forward to my gmail account" feature. That's how IT can be relevant: provide that system. You might prompt the business to reclassify some formerly sensitive data as rubbish they're allowed to play with, but then their fingerprints will be all over the corpse.

Uploading patient/client data to the cloud where a Google bot can read it is a breach of that patient/student's privacy. Blackberry email and the like can make email within the organization secure, and most teachers/doctors have the sense not to email sensitive documents to people outside the organization. However, most don't realize that emailing something to your gmail or uploading it to google docs is a problem. The mental model is still "this is private" even though Google is reading every word.
Maybe you've been subjected to more complete DLP systems than I have, but email "within the organization" is not and never will be "secure".

Every time I've seen customer demographic data emailed (although admittedly this hasn't been in the medical field), both the sender and the receiver have been employees (including myself) who weren't entitled to see that data. Organizations need to find more appropriate ways to collaborate, which don't needlessly expand the pool of people with access to sensitive data.

You seem to trust a pool of 100 people, even if they have acronyms following their names, more than you trust a search engine, to not share data in legally negligent ways. That seems ill-advised to me. If the Googlebot were generating lawsuits for breach of privacy we would have heard about them.

I don't think this sensitive customer data should be in Gmail, because I don't think it should be in any email system period.

Evernote, Dropbox, Google Docs. All have TOS and things like Apps can have contracts. Apps has a DoD clearance.

There is a legal distinction between subcontracting out services and sharing data. One that has no difference from paying for a service contact that allows a vendor to login and fix your db.

There are very few situations where EVERYTHING must be internal.

Google Apps is big in education, so "sharing" data under contract must be legal.

You're really fighting two mantras - 'if it's not broken, don't fix it' vs 'we must build against worst case everything'. The arguments generally come from IT support and legal, respectively.

Realistically things are in the middle. This isn't a surprise. IT shops have to balance current real risks, potential risks, future risks, etc. It's the overly used 'black swan' event in IT that causes problems. It costs $200k per potential problem, and we've got 40, but the business only provides $1M in budget. So the black swan will happen, the business will demand a solution, so now you've got 41 problems - because 2 surfaced while fixing the 1.

To take a step back, it's simply because consumer IT has innovated quicker than both enterprise IT and enterprise security to prevent the takeover. Trying to understand that is a more interesting question, which probably finds its roots in the blossoming technology adoption of a younger generation more willing to consume high tech goods. Eventually enterprises adopt consumer technology, or build really good walls.

This is certainly the case with schools too. At my high school, we are provided a username and password to access the school's computers, as well as our own personal storage space on the network. However, students (and teachers) want ways to work on files they have on the school network­ — it used to be that we would have to email the files to ourselves, but the network administrators have just recently unblocked access to Dropbox. People are realizing that there are websites like Google Drive that will let them access their work from anywhere and migrating away from the school-provided storage.

Last year, someone was able to find a vulnerability in the network in order to install Google Chrome and Firefox. Supposedly, the IT guys were furious — not just at being hacked, but that students were using software that wasn't approved by them. Students and teachers are wising up to what good software is for them, and those choices don't always align with what IT says we need.

It's not just about being hacked or using un-approved software.

For example, Google Chrome allows itself to be installed to a user account, bypassing administration requirements which may be that "vulnerability." The install is not particularly big, 50MB or so, so when Little Johnny Hacker does it it may not seem like a big deal. When 20,000 students install it that's almost 1TB, before we even consider them actually saving school work! (If you don't have 20,000 students in your school lets assume your IT resources and staff are appropriately scaled.)

You might ask, when you've got 20,000 people who want a piece of software why wouldn't you just make it available to them? So, let's say your school uses some web tools like Blackboard Learn and somewhere along the line--maybe in Chrome, maybe in BbLearn, maybe in Java, maybe somewhere else--there's a bug and students can't upload their homework to BbLearn with Chrome.

Now you've got 20,000 student freaking out and swarming the help desk trying to figure out what to do, teachers are upset they have to change their plans since it's not the students' fault, and IT is flustered because this is an emergency and not something they can research and test and find an appropriate solution for their environment.

And all this because, clearly, the students know "what good software is for them" and IT is just a bunch of old hacks who can't keep up.

When you work in any collaborative or networked environment some sacrifices will be made to fit everyone in. It's an IT department's job to figure out what technology will make the cut and what won't. Some of those decisions will be good, some will be bad, and some decisions won't actually be in the IT department's control. If you don't like a decision that was made (or wasn't made), you should talk to IT about it. They may tell you to bugger off, or they may make an exception for you or even launch an investigation to launch of complete solution.

Sadly in large companies with IT departments that have accountability and as such have internal costing to another department. Well in those sitauation it is often common for one department head to go behind official channels and outsource for a cheaper price. This sadly bypasses alot of security and other standards the company has. It's not new, and will happen again and again.

One example would be bank that had a website defaced around 12 or so years ago in protest to petrol prices. Turned out that the server was located in a server room with a dog running around in it and would be best described as a spare bedroom almost. The marketing department manager had organised that gem of a disaster. Was lucky as forensics upon that server indicated it had been hacked at least half a dozen times previously. So the defacement hacker had done that bank a realy big favour.

So your company can have the best and most excellent security standards in the World that are completely unbeatable. But it only takes one department head to outsource behind your back or for one individual with a BYOD or the like to plugs in and your open to a screwing.

Clouds are popular as for some reason people have been sold that there all uber secure in that all your worries are removed. They are not, shifting the storage elsewere not only opens up another access point publicly to potentual get at your data but the over comfortable attitude it installs will be inclined to make the clients not as secure as they should be.

If I was a Administrator and I was responsible for the data and liable to getting legaly shafted if there is a breach and the company used clouds and had a BYOD policy then I'd be very much underpaid and with that googling for some form of disclaimer you got every user to sign and every manager to sign. Just so I could sleep at night.

Remember this, when it comes to IT most users are like children and with that they will find a way to break it if one exists and failing that they will find a way.

Block everything website wise and add as an exception, as there realy isn't many websites that companies need you to access. If you want to access any other site then BYOD and network, just don't go driving on the internet in the name of your company. I often wonder if I was to set up a free porn site and then check what companies have employee's browsing it and then have a name and shame of the companies. But I feel that would be cruel upon poor employees with a porn addiction and with that I just can't do it as it would just get alot of people sacked and no company would take any heat from it.

WTF?
Sorry I realy don't understand your question!
In banks, especially those with large capital market or investment banking arms, you WILL risk losing your job if you try to work around corporate IT. It is basically a guilty-until-proven-otherwise perspective. I have seen it happen multiple times to front desk personnel.

That is also assuming you can, since many banks have super strict policy implementations which would necessitate greater than average technical know-how or investment to work around them.

Of course, there is a cost to this type of infrastructure. Whether you can dilute this cost to make it more accessible to ordinary companies by technical means alone, is something I suspect is not possible.

I remember at an old job on a stock trader's last day he emailed himself (from corporate email to gmail) a spreadsheet that contained proprietary models, client holdings, etc. That's a serious breach, and luckily traders are dumb enough to use corporate email to do this because if he used something like dropbox it probably would never have been caught. I don't like being restricted ever, but you can see why a company might try to block these cloud storage services to protect itself and its clients.
TBH, client details really should belong to both the firm and the traders, since at the end of the day those clients most likely will continue to execute trades with that trader regardless of what firm they work at. Back when I worked in finance, many traders I knew were hired based on the clients with whom they had a solid professional relationship.

The value of a trader to a firm is essentially their professional relationships with clients combined with the efficiencies and information provided by the firm itself. The trader needs information from the firm and his co-workers to effectively monetize his client relationships, but those relationships really are his/hers at the end of the day. It's not like a trader can leave a firm and some other trader can pick up those relationships right where the other trader left them off. They can try of course, but the relationships are likely to move from firm to firm with that trader.

The spreadsheet is also dubious grey area. Yes, it may be proprietary information created by the trader while at that firm, but it is just as likely to have been created by that trader before he joined the firm that he brought with him when he joined. The only thing that changes when a trader joins a firm is that he ceases to use inputs from the economists and analysts at his previous firm and now begins using the figures from the economists and analysts at his new firm. Proprietary models often are created by a trader and intelligible to that trader and only that trader, unless they happen to have trained a junior trader to understand the ins and outs of their own model.

I was one of the analysts myself and every single model created by any senior analyst was reused by their junior analysts, but was often scrapped anytime a new senior analyst who joined the firm to replace the previous senior analyst. When you have your name and reputation on the model and the investment advice, the tendency is to do a big rewrite.

I wonder what percentage of Shadow IT practices are due to organizations bending over backwards to appear PCI compliant.
You need two networks: one internal without any Internet connection and computers with no WiFi and no USB.

Make people work on their workstation, connected to the internal network and let them use their other computer / laptop to search the Web.

I can name at least one very important chip-designing company that is worth $$$ bn that used to work this way (don't know where they're at now).

I am not sure partitioning networks the way the military does is going to work very well. What is going to happen when a mid-level manager has a meeting, he is running late, and he just needs to get his powerpoint set out of the internal network? He's either going to miss that meeting or fail to close the deal, at which point the policy is getting in the way of business (and will therefore be short-lived), or he's going to find that one crack that lets him get some data out (more likely).

The reason red/black networks (can potentially) work in military environments is that there is a (somewhat) uniform notion of classification in the military; in the business world, there is no such thing. What is needed is something more distributed, like a system that automatically encrypts documents so that uploading those documents to some Internet service is not so hazardous. Give employees smartcards that are easily carried around and easy to use, perhaps combining those smartcards with a thumb drive that contains whatever software they need to decrypt their documents on any computer. The security will not be perfect, but this is not a situation that requires perfection, only improvement.

I work at a facility where all web browsing must be done through a remote desktop session to a server connected to the exterior network, which is reimaged regularly.

Unfortunately they don't keep software fully up to date on the remote desktop server, so the security benefits are lessened. But malicious websites have no way of stealing your secret files.

Unless there is a bug in your remote desktop client that can be exploited by a compromised server...
I interned at an outfit like that a long time ago. It sucked then and would never work now in the age of smart devices. Unless of course these devices were banned from, or confiscated on entry to, the workpla^H^H gulag.
I don't think I'd want to work in a place like that.