Never waste a good outage

Much of the world woke up this morning to TWO massive IT system outages, affecting business and critical infrastructure in many countries, including the US, Europe, Asia, New Zealand, UK, Australia and all over the world.

The outage is affecting millions of users and services, bringing down services like;

• Major banks,
• Payment systems,
• Transportation systems,
• Telecommunications,
• Mainstream media outlets,
• Etc.

Millions of Windows machines booted up this morning into the “Blue Screen Of Death” (BSOD), creating the largest systems failure in history.

Note: This problem has not affected any Unix or Linux based machines (lol).

To be fair, it really is not that hard to crash Windows, but unfortunately we can’t blame Windows for the outage (this time).

The issue was allegedly caused by a buggy update pushed by the U.S. based cybersecurity company called CrowdStrike (interesting name).

In the interest of never wasting a good crisis, I decided to write this article to provide a remedy for our clients, and as a reminder of the importance of decentralization and the use of open source systems, especially to protect critical infrastructure.

Open source.

Situations like this is the exact reason we promote and encourage the use of open source systems such as Linux, ClamAV, and other robust Libre software.

When a system is open source, developers and cybersecurity experts pick up on bugs and release patches much more quickly than in the case of closed source or proprietary software.

This is obvious now, but is something I’ve been talking about for years.

Decentralisation.

Decentralisation means to use systems that don’t rely on centralised services. Cloud systems are inherently centralised, and this process leads to single points of failure.

With a single point of failure, and a large blast radius, when companies using closed source systems fail, they fail big.

So what’s the solution.

For now, many companies and media outlets are trying to patch the immediate problem The solution appears to be doing the following:

Workaround steps for individual host machines:

• Reboot the host to give it the opportunity to download the reverted channel file. If the host crashes again, then;
• Boot Windows into Safe Mode or the Windows Recovery Environment
• Navigate to the C:WindowsSystem32driversCrowdStrike directory
• Locate the file matching “C-00000291*.sys, and delete it
• Boot the host normally

Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround steps for public cloud or similar environments:

• Detach the operating system disk volume from the impacted virtual server
• Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
• Attach/mount the volume to a new virtual server
• Navigate to the C:WindowsSystem32driversCrowdStrike directory
• Locate the file matching “C-00000291*.sys, and delete it
• Detach the volume from the new virtual server
• Reattach the fixed volume to the impacted virtual server

Long term, we recommend switching to Linux and other open source systems.

We offer a “Switch to Linux” service for individual users as well as larger operations such as businesses, corporate and government organizations.

Other services we offer to prevent this sort of thing from happening include;

– Linux services: Installation, implementation, integration, training and support.
– Linux computers: We sell Linux computers, including laptops, desktops and notebooks.
– IT support: We have IT support service level agreement retainer options available which include end point protection, disaster recovery, and remote desktop / server support.