When Employees Use Software That IT Hasn’t Approved

I see this a lot in consulting. When a new CIO (or CEO or other C level) arrives, they want to make their mark with a digital transformation intiative. This usually just means that the new C level employee is coming into a medium to large business and would like to add a bullet point to their resume and get that new shiny object everyone is talking about. Tableau, Salesforce, Data lakes, blockchain, ERP, Identity Management and &quot;cloud&quot; projects are often the result. It seems to also stem from the new C level employee having a close relationship with a sales rep&#x2F;partner&#x2F;C level employee at the vendor. Left a project a couple years ago that had Hadoop interfaces from every system. The user count of all this data? exactly 0.<p>One somewhat disturbing trend I&#x27;ve seen at some of the largest corporations- cut&#x2F;outsource IT support staff to near egregiously low levels to &quot;save money&quot;. At the same time kick off 7-9 figure ERP&#x2F;consulting projects that at best provide fractional value to the organization.<p>Of course there are counterpoints to this. One of Houston&#x27;s major pipeline operators pulled off a digital transformation and actually ended up with well designed, highly integrated and easily maintained systems. It took about 5-7 years and had a few reboots, but it eventually landed. That brings me to my final point. These projects often have a timeline that is divorced from reality. Whatever time frame you think a major IT project will take. Double it. twice, then add 50% and you are close. It also seems that C level folks are hesitant to hire boutique&#x2F;small shops that have industry experience and years of experience in favor of big consulting. Nobody every gets fired for hiring Accenture&#x2F;Deloitte&#x2F;PwC. What usually happens in the non trivial niches is that these big shops sleeve the boutiques through them to get things done...

This is exactly extremely common. In my company there is this constant battle about the devs having admin rights on their machines. We need admin rights to do our job. We have had dozens of meetings explaining the situation but IT can’t come up with a solution so the devs go around security because they have no alternative if they want to finish their work . Same with Dropbox. They block it but we have suppliers who use Dropbox. So the result is that people download confidential files from Dropbox on their home computers or phones and transfer them to their work machines.<p>In my view security shouldn’t be isolated at corporate headquarters but they should be close to the end users so they see what users need to do and help them to balance security with getting things done. They can’t just block stuff without providing alternatives or they will either hurt the business or they will be circumvented.

Let&#x27;s ignore the SaaS security issues for a second. When IT says &quot;No&quot; it&#x27;s not like the area asking is going to go away and not try to solve their problem. Organizations are going to find ways to solve their issues and IT can either help from the beginning or help clean up the mess later. I try to take the stance of offering the right solution and a lot of the times a now solution at the same time. There is no saying &quot;No&quot; in the long term, either help them now or get stuck with the shadow solution the magic macro guy cobbled together that became a critical business function.

I still shudder thinking about my time working as a developer on corporate IT locked down IBM leased laptops. Every time I did npm install I needed to request admin access to Windows which took 2-3 hours to action by IBM team sitting on the other side of the world in India.<p>One day a grey beard took pity on me and installed a Linux VM where I was admin, copied the security certs from the Windows host and I could access all corporate resources at my leisure. Never logged a single IT Helpdesk ticket after that.

Our IT security department was incentivized to deny everything from new tools to new internal applications.<p>We had an outside firm making security decisions and if there were any security issues it would end up being on them. So as long as they did not allow us to release any products and or install any software they could not be held responsible.<p>I made friends with a lower level contractor who told me off the record to use my judgement on what to install to get the job done, because the security department would never approve anything new unless directly instructed to by the CEO.<p>Fast forward three months, there was a major security flaw on our website (also built with outsourced labor) which allowed anyone to access private data without a login.<p>A few of us had reported to the security department that the code running the website was so poorly written that the odds of being insecure were close to 100 percent. We suggested upgrading the website and rewriting the code, and management was on board with this but security department refused to allow us to use any new frameworks since they were not approved. Of course in a matter of a few months the site was hacked and millions were spent as a result.<p>I quit this job after we were unable to release several products after a year even though we jumped through every hoop we needed to. That department killed all innovation.

Joining Google was an eye-opener for me on this. Was the first time I encountered an IT department (TechStop) that didn&#x27;t act like a police force and instead had your back, helping you get where you needed to be. Was always the first thing I would show guests on a tour of the campus.

Its the user who downloaded a program they &quot;needed&quot; which had malware which sent out a lot of spam email because this was a user that did announcements which basically got an e-mail server listed on blacklists that creates these IT policies.<p>You want to treat people like responsible adults, but they aren&#x27;t the ones who have to deal with the fallout. Developers know the score for the most part, so full privileges are expected with the caveat, if it all goes bad, we are wiping the machine[1], not doing a recovery.<p>IT dreads the moment we are called to account for something some user decided they needed to do.<p>1) most developers understand backup tools and code control - those that don&#x27;t, well...... with great power comes great responsibility

<i>&quot;Soon enough the CIO sniffed out the project and called her in to a disciplinary council.&quot;</i><p>Somebody has apparently lost touch with who the customer for IT is.

Security training focuses way to much on email phishing and not enough on this kind of stuff. Actually getting your work done, managing your own computer. Of course people can&#x27;t be trusted if they havem&#x27;t been trained. How to handle USB drives. What and from where you can download and run programs. What actually IS a program and what isn&#x27;t. Many of us learned this the hard way by playing lots of cracked games in the 90s. But not everyone did that.<p>Try explaining to a non-technical person how how a desktop background image isn&#x27;t a program so it&#x27;s basically safe to grab from anywhere, while a screen saver is <i>definitely</i> a program and usually unsafe to get from most places, and a word document is <i>some times</i> a program that might eat your computer. Training could involve things like &quot;which of these 5 webbpages would you consider it safe to download and run executable from&quot;?<p>Having too cumbersome rules around security just means it&#x27;s ignored or circumvented, increasing risks.

This brings back a memory. The only time I was fired &quot;for cause&quot;. Summer after my freshman year at college, I was temping and got an assignment doing real estate purchase comps with a company in the East Bay. At the time, there were laser printers, but often printing sucked up CPU time and let&#x27;s just say multitasking was still not a widespread thing.<p>I found myself tired of sitting around. I found a TSR &#x2F; print spooler that would use RAM and offload the process of printing. This allowed me to keep working. My productivity (as a temp) was higher than many others including the person I was &quot;reporting to&quot; at the company.<p>They found the print spooler, labeled it &quot;unapproved software&quot;, and I was walked out the door.<p>The funny thing is, a friend at the time (and I didn&#x27;t realize it) was higher up in the management. He reached out to me on a multi-line BBS that was popular in the area and offered me a full time job a few days later. I was in school and obviously declined.<p>Working the rest of the summer for a Chemical Engineer in Martinez&#x2F;Benecia ended up being incredibly more interesting. So it was a net win.

This thread hits home. I switched jobs a few years ago because the IT policies on workstations were being ratcheted down to make my job as a developer difficult to impossible.<p>Now, the company I work for, ostensibly a _software_ company, got its ISO certifications, which meant policies and procedures that make developing hard or impossible again.<p>How does a software business _successfully_ implement stringent access controls while still allowing for efficient software development? I&#x27;d like to see&#x2F;hear what works.

IT is a service to the rest of the company. If you don&#x27;t approach it with a servant&#x27;s heart, people will go find their own solutions without you and you&#x27;ll be part of the cleanup crew.

I just witnessed a very similar situation, on a smaller scale, but there are many of these in my company, and they add up.<p>Boss: &quot;We need access to the database of our primary application that you wrote for us so that we can pull the data into this new tool to track progress.&quot;<p>IT: &quot;No. Not only can we not give you access to YOUR data in YOUR application that we wrote on YOUR dime, we will not allow you to have this new application written by someone who isn&#x27;t in our group. If you wanted something like this, why didn&#x27;t you just ask us? We would have written this for you.&quot;<p>Boss: &quot;We had a meeting about this over a year and a half ago, and you told me that you didn&#x27;t even have the time to discuss it further.&quot;<p>IT: &quot;... Well, we&#x27;re still not going to let you do this.&quot;<p>IT is effectively holding the rest of the company hostage, and the corporate technical debt is becoming epic. So skunkworks solutions will continue to be developed.

Its the same dilemma companies face with their legal team. The &quot;safest&quot; thing to is nothing at all so sometimes the overabundance of caution hamstrings business growth.<p>That&#x27;s what the CEO is there to figure out.

The ideal IT team is one that proactively learns the needs of others in the business and works with them to solve problems. It&#x27;s no wonder that so many companies end up with shadow IT when so many IT teams are just people who tell you &quot;no&quot; whenever you ask for something. Doing it right is harder in the near-term, but much easier in the long-term as you&#x27;re not putting out so many fires or going to &quot;disciplinary council&quot; meetings.

It sounds like the author recommends embracing the Agile philosophy of letting your teams choose their tools, then working with IT&#x2F;Sec to make sure implementation is sound. I like that philosophy.

As a born and bread corporate (mostly banks) corporate IT guy, I used to frown upon this behavior. Then I got one of my bigger career breaks because the finance team went behind IT, bought a software and installed it in a machine which kept under their desk. They further hired people from an IT service company to configure the machine.<p>The configuration was so bad that it exposed the company&#x27;s network to whole wide world. Google contacted the company and after searching high and low IT security managed to track the pc down and take it away. Finance team promised to hire someone with skillset required to run the software in a closed environment. And that&#x27;s how I ended up getting my job.

This is my current situation. Being a SWE, IT and security are always putting out fires with networks or upper echelon cybersecurity violation complaints (mostly people downloading software without authorization). They have very little time, almost none for investigating new software, and all software must be installed by them. End of the day, nothing gets done on our work computers. I once waited two months for them to say no for a piece of solution we as the team approved. It&#x27;s absolutely frustrating.

I&#x27;ve been amused by VMWare being on the strictly-enforced official software list, and the VM being considered data. Nothing in the VM counts as software! It&#x27;s not even being sneaky. Official policy is that the VM is data.

&gt; The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP. The CIO was silent. Then the CEO asked the VP, “I’ve known you for ten years. You don’t seem like someone who would do something to harm the company. Why did you do this?” The VP hit right back: “Since I started this digital customer acquisition program, we’ve increased revenue $1M per month. Before we were losing revenue. If you want, I can shut it down right now. What do you want me to do?”<p>Maybe not for this particular project, but another interpretation of that is &quot;who cares about security if we&#x27;re making money&quot; which is a very dangerous argument as well.

How many years since <i>The Phoenix Project</i> and this conversation has barely moved an inch?<p>CIO probably wins this battle and gets the VP fired, but will be mystified when they&#x27;re reporting to the CFO or a Chief Digital Officer when it happens 3 more times by the end of the year.

Something that&#x27;s crossed my mind is John Gall&#x27;s observation that complex systems operate in failure mode 100% of the time. I understand &quot;failure mode&quot; to mean that built-in guards have been bypassed in order to enable the system to do <i>anything at all</i>. Germane to this thread, the &quot;guards&quot; are IT approvals.<p>I suspect that if a business is complex enough to have IT policy, that policy is always being bypassed in some way, at any given time. Somebody is using unofficial software, or using official software in an unofficial way.

To me it reads like this - VP didn&#x27;t care about the consequences of utilizing their solution and didn&#x27;t care about IT; they simply wanted their stuff done, without acknowedgling prioratization of tasks.<p>The proper way this could have been resolved is by VP utilizing people&#x27;s skills they&#x27;ve hired. Does this solution look good and will accomplish the task that was prioritized? Excellent! Pass it to IT to evaluate. If the task has specification - excellent, have somebody in IT look for a product that ticks all the boxes and let&#x27;s choose it together.

Does your IT team use key loggers or other employee monitoring software at your company? I hear some big trillion dollar companies do this. Is that true?

I work in an IT organization &amp; I see (in the sense of witness) both sides of this. We are over-tasked and under-resourced and new projects&#x2F;ideas&#x2F;initiatives that come in the door go into a backlog of requests. So I see business&#x2F;end users signing up on their own for SAAS solutions to solve their problems.

&gt; If you don’t think this is happening in your organization, think again<p>That story probably never happened anyway. But the essence of the article is very true. I never have been in a corp where IT enforces 100% conformity anyway (apart from medical industry).<p>Sure, there are actual successful attacks, but that is mostly not the fault of unsanctioned programs.<p>But there are systems where people should not just start to use any system, because information gets lost on the way. That would include CRM and ERP in my opinion. That a company can exist without a CRM is questionable to begin with and solutions are plentiful. If they did not have anything like that...<p>If the story were true, it would not be the fault of Chief Input&#x2F;Output.

Well, that cleared that up then! Gosh I had no idea that the solution would be so simple.<p>It does shock me that the people who&#x27;ve had their whole infrastructure compromised and held to ransom by viruses and the people who&#x27;ve been held over a barrel by suppliers or had vast amounts of money burned by being locked into a dozen vendor contracts for the same service are so silly and hysterical about it when the solution is as simple as &quot;identify when you need to be best in class and stay small everywhere else&quot;.

Hehe, if you think this is nuts, come to pharma. We can&#x27;t do jack shit with our machines. If you so much as change the time on your machine, that is a &#x27;data integrity breach&#x27;, and if your actions are determined to be malicious it can result in a firing.

I just wish I had a decent computer at work that didn&#x27;t have 3rd party antivirus that would just slow the software to a crawl.

Isn&#x27;t IT something from the past? I would expect people knowing how to use a computer and what they need to do their job.

There&#x27;s a difference between using software that IT has approved and shipping customer data outside of the company.

how about trust your staff.

In our company people just started using free slack en mass, boycotting the horrible IT approved Skype for business. When it was discovered that thousands of employees were using slack, the CTO had to step in and tell IT to fuck off, and started paying for the full version.

&gt; The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP. The CIO was silent. Then the CEO asked the VP, “I’ve known you for ten years. You don’t seem like someone who would do something to harm the company. Why did you do this?” The VP hit right back: “Since I started this digital customer acquisition program, we’ve increased revenue $1M per month. Before we were losing revenue. If you want, I can shut it down right now. What do you want me to do?”<p>Shut it down right now and ask the VP to tender their resignation. Any company doing a 3-year SAP implementation is a very large company. That $1M in additional revenue pales in comparison to the risk introduced by sharing company or personal customer data with a vendor who has not passed the required security auditing. Data is no longer a thing to be thrown around in search of additional revenue and &quot;but I made money&quot; or &quot;I had to because IT is slow&quot; is not a post hoc rationalization for the behavior.<p>Regardless of the merits of large enterprises acting this way, this is a VP who clearly cannot function within the enhanced risk-controlled environment of one and should find a position with a smaller company where they have more freedom to pursue personal initiatives at the VP-level. Those companies exist. Go find one.