🌲 treerockets blog

LOAD "DIGITAL NEGLECT",8,1

The German term "Digitale Verwahrlosung" is hard to translate, but means digital neglect. It's not like that this is an actual common German saying, however it - to me - describes the issue the best. We are going to experience a transition from seemingly ancient technology, to modern systems. But who can do it? The necessary skill-set for this is owned by very few; and this "crowd" is shrinking.

We are steering towards a huge problem - it's already happening but not on as large of a scale. yet. Worldwide - many systems are run on "ancient" technology. I'm not only speaking about the 2003 Server that is still in countless (German and others) server rooms - running and hosting real live data. The problem is spread much wider. Look at old telephone systems in many offices. More often than you'd like to see, this is still hardwired technology from the 80s and 90s. Hundreds and thousands of wires, hard-wiring internal number 843 to the actual phone line, handled by some PBX that puts the call though to the correct desk of Hans.

This may be a relative extreme example but it emphasizes the issue quite well. There is only a few people who can - first of all properly maintain this - and even less people who can properly migrate this to something more modern. What if Hans is moving office and wants to take his internal number with him? Can someone configure this in the company? For migrating a phone system there may be workarounds, worst case; running 2 systems at the same time for a short time, for other systems; this is not possible or very complicated. Downtime is expensive, panned or accidental.

It's only a few years back - and were today in 2022, I spoke to a customer on the phone about a firewall issue on their "most important system". 2 physical - bare-metal, windows 2000 servers in a rack with 4 dedicated power-supplies, essentially doing the same job - one is a backup in case the other breaks. They can not be offline for even a second.

Luckily this machine was - except for very strict rules - more or less not accessible via network - however they where unable to shut this machine down, take it offline or even migrate it. They didn't dare to switch one off and hope that the other kicks in. They had an emergency plan, but nobody was ready to take action. If it would not work, it could cost millions per hour - and would affect a "lot" of people.

The person I spoke with, was the only one in the company capable of managing this machine since it's so specific and so on the verge of breakdown... We spoke, he described how important it is for him to document EVERYTHING. But that is not going to be the normality. I experienced this - so I it's safe to assume that there is others like it. Thousands. Millions.

What is going to happen to these machines? In many many environments the current admin is keeping them alive - avoiding migration - until retirement... and then? There is nobody who has experience in this.

CNC milling machines with some ancient MS-DOS on it running G-codes in a metal-factory. They don't have time to take them offline and stop production. They also can't afford to buy a new one.

In some scenarios there's going to be a handful of people specialized in ... let's say migrating 2003 servers including all services to something modern... but not always.

I recently spoke to an entry level admin, just of the University. He was dumped on an environment with a 2003 Server that did everything. DHCP, SMB (v1!), DNS, AD and a whole bunch of other things he didn't even know about. He was trying to migrate this to a 2019 machine - and ran into problems. Of course, this is not an entry level job. But can a new hire afford to say "This is not an entry level job, this needs a bunch of specialists"? Unlikely.

The more special this gets, the more vulnerable these systems are, the fewer specialists are going to be there, the more expensive it'll be. So it'll stay until it breaks. And then? ...

Looking at the current situation in IT, I experience horrifying things, I consider as the absolute basics. However ... speaking to someone who is supposedly in charge with clearly no clue. I'm not even blaming the person who is in charge, I'm blaming everyone around it. Often these people slip into these positions "because they know stuff" or they simply can't afford to say "I can't do this, this was not in the job description". It happened to me. It happened to friends.

But - I'm of a generation that grew up with computers that didn't do anything unless you wrote a program yourself; telling it what to do. Be it a Commodore 64, ZX81, a large PDP 1 ... We - I and others - have a completely different approach to technology like this. I can figure out things in a very different way than generations after me. I grew into "figuring it out".

When I buy a new laptop I don't even boot it first - I stick in a USB and install the OS of my choice. My neighbors son of 17 years, has no clue what I'm talking about. And he's using his laptop every day.

So what's the solution?

The solution must be to value what the current IT expertise can provide. Cherish the admins, pay them appropriately, respect them and encourage them to migrate, update, upgrade IN TIME. Hardware, Software, Physical security, Machinery. The exact same is for support staff. There are currently a whole bunch of people out there who still have the required experience to do all this. This generation however, will not be there forever.

Laugh at me all you want, I know of more than a enough environments that are "kept old" to make sure the current admin can keep it's job as long as possible and is not replaced by some young student or some remote-admin team in Bangladesh. And he's already unterpaid as is. It's not the fault of this admin. It's the fault of the company - and there are many cases like this.

#IT #security