In light of the recent global outage caused by the CrowdStrike Falcon Sensor update, businesses and individuals must follow a structured recovery process. This guide is designed to help the IT community quickly recover affected machines, with insights on how mature partners like FUJIFILM CodeBlue Australia can assist in mitigating and resolving such incidents.

Background

On Thursday, July 18, 2024, CrowdStrike reported widespread BSODs on Windows hosts due to a defective Falcon Sensor update. The affected update began propagating at 04:09 UTC and was pulled by CrowdStrike by 05:27 AM UTC. Machines booted up after this time should not be affected.

Recovery Steps

There are two main options to recover a Windows host that has blue screened:

1. Restore from a snapshot prior to 04:09 UTC
2. Recover the machine using the following steps:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
   For physical devices, restart and press F8 (or relevant key) before Windows starts.
   For virtual devices, use your cloud provider’s recovery tools.
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
   Locate the file matching “C-00000291*.sys” and delete it.
3. Boot the host normally.
   Note: The steps can vary depending on whether the Windows host is a physical or virtual device, and steps for cloud-hosted virtual machines vary depending on the cloud provider.

If the machine is running BitLocker or other disk encryption software, additional steps may be required to boot into a Windows Recovery Environment.**

BitLocker recovery-related KBs:

• BitLocker recovery in Microsoft Azure
• BitLocker recovery in Microsoft environments using SCCM
• BitLocker recovery in Microsoft environments using Active Directory and GPOs
• BitLocker recovery in Microsoft environments using Ivanti Endpoint Manager

Recovery Steps for Cloud Services

1. Microsoft Azure  
   1. Azure Status page 
   2. Recovery Options: 
      1. Restart the affected VMs: 
         1. Using the Azure Portal, attempt ‘Restart’ on affected VMs. 
         2. Using the Azure CLI or Azure Shell: `az vm restart –name <vm-name> –resource-group <resource-group>`
            
      2. Restore from Backup: 
         1. If possible, restore from a backup prior to 04:09 UTC. 
         2. Follow instructions for [restoring Azure VM data](https://learn.microsoft.com/en-us/azure/backup/backup-azure-arm-restore-vms).
      3. Attach OS Disk to Repair VM: 
         1. Follow [these instructions](https://learn.microsoft.com/en-us/azure/virtual-machines/troubleshoot-recovery-disks) to delete the problematic file on the attached disk.
2. Amazon Web Services (AWS)  
   1. AWS Status Page 
   2. Recovery Options: 
      1. Reboot the Instance: 
         1. Attempt to update the CrowdStrike Falcon agent to a previous healthy version through a reboot.
      2. Delete the Faulty File: 
         1. Create a snapshot of the EBS root volume of the affected instance. 
         2. Attach the EBS volume to a new instance and delete “C-00000291*.sys” from the attached volume. 
         3. Create an AMI from the updated snapshot and replace the root volume of the original instance.
      3. Restore from Backup: 
         1. Relaunch the instance from a snapshot or image taken before 04:09 UTC.
3. Google Cloud Platform (GCP) 
   1. GCP Status Page 
   2. Recovery Options: 
      1. Manual Patch: 
         1. Follow CrowdStrike’s support portal instructions. 
         2. Attach the boot disk of the affected VM to a rescue VM. 
         3. Delete “C-00000291*.sys” from the attached disk.

Microsoft’s Automated Recovery Tool

As a follow-up to the CrowdStrike Falcon agent issue impacting Windows clients and servers, Microsoft has released an updated recovery tool with two repair options to help IT admins expedite the repair process. The signed Microsoft Recovery Tool can be found in the Microsoft Download Center:

Microsoft Recovery Tool.

This tool includes detailed recovery steps for Windows clients, servers, and OS’s hosted on Hyper-V. The two repair options are as follows:

Recover from WinPE – This option produces boot media that will help facilitate the device repair.

Recover from Safe Mode – This option produces boot media so impacted devices can boot into safe mode. The user can then login using an account with local admin privileges and run the remediation steps.

For the detailed steps on Microsoft’s recovery tool, refer to this article New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints – Microsoft Community Hub