Crowdstrike must come clean

Security experts are still in the dark about the underlying causes behind the massive failure of business Windows systems running Crowdstrike’s Falcon security software. The circumstances under which this could happen require clarification from the manufacturer.

Millions of computers showed a blue screen last Friday, indicating that they could not boot. During the day, many organizations managed to restore the systems. They slowly became operational again. According to an estimate by Microsoft, 8.5 million devices worldwide were affected. That is less than 1 percent of all devices running Microsoft Windows. A small percentage, but with major consequences.

Sandbox

Although it is clear where things went wrong, software supplier Crowdstrike has not yet provided an explanation for the fact that a programming error in a routine update went unnoticed and slipped through the testing phase. The question is whether the correct testing procedures were followed before the rollout of this widely used software took place. Normally, a new piece of software is first placed in a protected environment (sandbox) to see how the code behaves.

This isolation prevents the rest of the system from being damaged or even failing completely. The frequency with which software companies release updates can lead to testing procedures being rushed. Reuters news agency quotes various security experts on this point.

Crowdstrike reports in a blog that it is conducting a thorough root cause analysis to determine how this programming error in the logic arose. The Texas company is also looking into whether fundamental improvements or a better workflow are needed to prevent this type of error in the future.

The fact that the problem could quickly become so enormous is due to the automatic distribution of updates. If they contain an error, all computers that are currently switched on or in active mode are immediately affected.

The incorrect update of the Falcon Sensor program was distributed last Friday at 04:09 local time (UTC). The error affected computers that were switched on all over the world for 78 minutes. According to some experts, the brain of Windows, the kernel, failed, resulting in a complete crash. Computers could no longer boot.

Channel file

The updated configuration files are of the type ‘channel file’. Such updates occur several times a day in response to new tactics, techniques and methods of attackers. These files contain data to neutralize cyber threats. This is certainly not a new process. Crowdstrike has been using the same architecture since the launch of Falcon.

Although the channel files (in this case number 291) end with the extension .sys, Crowdstrike claims they are not kernel drivers. The error also does not relate to zero bytes in a channel file, according to the company. It is denied that a Null pointer from the memory-unsafe language C++ was the culprit.

In any case, Crowdstrike still has a lot of explaining to do to restore trust. Because Crowdstrike has many (large) business customers, the incorrect configuration update could lead to one of the most widespread technical disasters. It is not impossible that such problems will recur in the future. Incidentally, the outage was not the result of a cyber attack.

Emergency plan

Companies must make emergency plans when systems fail. They must practice with them. Due to the intertwining of processes in the digital ecosystem, everyone can experience the consequences of a cyber incident such as last Friday when flights had to be cancelled en masse and banks, shops and hospitals were also affected.

The speed at which the problems were resolved varied greatly per company and per department. Because many affected machines could not start up, support teams had to come by in person. The implementation of the recovery software, which has to be done manually, takes a lot of work and time. Crowdstrike’s workaround has been effective.

Several cybersecurity organizations warned of an increase in phishing. Cybercriminals tried to take advantage of the situation by supposedly offering solutions.