P Is For Patching
I wanted to talk about patching today. And how it may appear that some organizations are “struggling” with it, which is an unfair statement to make, in my opinion. The very word “struggling” makes it sound as though they can’t get themselves organized to patch!! When in truth, the reality is quite a bit more complicated.
The most common issue I see is systems that just cannot be patched because they use functionalities of the OS that will somehow change when patching, and that will ultimately break the application. You patch and underlying OS API library changes, and your application no longer works. You call the application vendor and they tell you that their patch for the OS patch only comes out the following month. Now what do you do? You have no choice but to reverse; or choose not to use that application. A decision that must be made based on the importance of the application itself in the organization – do I stay with an unpatched system for a month or 2, or do I turn off that application for a month or 2? I’m quite certain most IT people will choose the first option then cross their fingers and hope for the best.
Another issue is the sheer quantity of machines to babysit.
You’re not just patching Windows.
You have tens of applications running on a workstation, all needing updates, almost every month. The monthly Patch Tuesday cycle started by Microsoft? It has become quite the zeitgeist in our industry. There’s just so much patching that needs to be done all the time that there are now even companies making a living with software to oversee patching, and patching alone – patch management software. Which, btw (and ironically), also requires patching!
Try patching a server, which is being used by your entire company. Put yourself in the shoes of the IT manager in charge of such an ungrateful task. It’s time to patch, and you start sweating – what if the server breaks? Sometimes they won’t even come up anymore. Oh, wait, that server can’t be touched during business hours, so the patches need to be applied at night and possibly at 2AM on Sunday. And if it does break, now you’re stuck spending your entire Sunday rebuilding a server you were trying to improve but instead ending up breaking. It’s like taking a vitamin to feel better and instead dying of hypervitaminosis. Yes, that’s a thing!
Enter virtualization.
Now not only do you need to update your servers and workstations (VDIs to be precise). But you may be tempted to keep up also with the virtualization software, be it VMWare or OpenStack or something else; and with the underlying OS (most likely Ubuntu or some other flavor of Linux). And if you’re into open source, the disks may be managed by something like ceph; another software that, in theory, you should be updating. Why? Because what’s the point of updating the dashboard in your car if you never maintain your wheels? What’s the point of patching Windows if your underlying infrastructure is vulnerable and someone could take down your entire virtualization environment? Sure, this part of patching doesn’t happen often but not for lack of wanting. Ubuntu, for example, releases plenty of updates, almost on a daily basis, in fact. But the last time I updated my Ubuntu workstation, it broke my video card, necessitating some sweating to troubleshoot and restore it. Do I want to risk that with my virtualization environment? Certainly not if my entire company depends on it. You may be able to move all the VMs out of a physical server, update the base software, and move the VMs back. A lengthy but feasible process. And I stress, lengthy.
You see now that patching isn’t as easy as we sometimes make it sound.
I myself in the past have made the harsh statement that people don’t patch because they’re lazy. While I’m sure a few of us fall within that category, I have, over time, come to a revised understanding. Patching is a tough thing to do, a minefield that can cause many different problems, which all lead to losing servers and workstations, and interminable long hours to rebuild those machines.
Patching can cause problems.
And that’s typically why IT folks are wary of doing it, and too often. There’s simply far too much to patch, every day, and we can’t spend our lives on this singular task. Therefore, we procrastinate despite our best intentions.
After all, it’s only normal for all us to postpone doing the things which give us a stomachache.