Crowdstrike Outage Blue Screen of Death (BSOD)

Summary of the Outage and Its Impact

One of the recurring issues reported was the Blue Screen of Death (BSOD) on Microsoft Windows Server instances running on Azure, exacerbated by CrowdStrike Falcon updates. This BSOD led to boot loops, significantly disrupting operations. Users reported needing multiple VM restarts to resolve the issue temporarily, but more robust solutions were necessary for long-term stability.

One of the recurring issues reported was the Blue Screen of Death (BSOD) on Microsoft Windows Server instances running on Azure. The BSOD, also known as a stop error, forces the VM to restart and can lead to boot loops, significantly disrupting operations.

The Script

Here’s the PowerShell script I used to resolve my VM boot loop issue:

# Import Azure module
Import-Module Az

# Prompt user for inputs
$subscriptionId = Read-Host "Enter your Azure Subscription ID"
$resourceGroupName = Read-Host "Enter the Resource Group Name"
$vmName = Read-Host "Enter the VM Name"
$rescueUsername = Read-Host "Enter the local user name for the Rescue VM"
$rescuePassword = Read-Host "Enter the password for the Rescue VM" -AsSecureString

$RepairRunId = "win-crowdstrike-fix-bootloop"  # Do not change this value
$Timeout = 600  # Timeout for waiting in seconds
$Interval = 120  # Interval for checking VM status in seconds

# Connect to Azure account
Connect-AzAccount -SubscriptionId $subscriptionId

# Step 1: Create rescue VM
Write-Output "Creating Rescue VM..."
az vm repair create -g $resourceGroupName -n $vmName --verbose

# Wait for the rescue VM to be up
$startTime = Get-Date
while ((Get-Date) - $startTime).TotalSeconds -lt $Timeout) {
    Start-Sleep -Seconds $Interval
    $vmStatus = (Get-AzVM -ResourceGroupName $resourceGroupName -Name "$vmName-Repair").Statuses | Where-Object { $_.Code -like "PowerState/*" } | Select-Object -ExpandProperty Code
    if ($vmStatus -eq "PowerState/running") {
        Write-Output "Rescue VM is up and running."
        break
    } else {
        Write-Output "Waiting for the Rescue VM to be ready..."
    }
}
if ($vmStatus -ne "PowerState/running") {
    Write-Output "Timeout reached. Rescue VM is not up."
    exit
}

# Step 2: Run the mitigation script on the Rescue VM
Write-Output "Running mitigation script on the Rescue VM..."
az vm repair run -g $resourceGroupName -n $vmName --run-id $RepairRunId --run-on-repair --verbose

# Step 3: Restore the fixed OS disk to the original VM
Write-Output "Restoring fixed OS disk to the original VM..."
az vm repair restore -g $resourceGroupName -n $vmName --verbose

Write-Output "The original VM has been restored successfully. You may delete the rescue VM if desired."

Instructions to Run the Script

1. Ensure you have Azure CLI and Azure PowerShell Module installed.
2. Save the script to a .ps1 file, for example, RepairAzureVM.ps1.
3. Open PowerShell with elevated permissions (Run as Administrator).
4. Navigate to the directory where you saved the script.
5. Run the script using the following command:.\RepairAzureVM.ps1
6. Follow the prompts to enter the required information:
   • Azure Subscription ID
   • Resource Group Name
   • VM Name
   • Local user name for the Rescue VM
   • Password for the Rescue VM

Running into issues with Azure VMs can be daunting, especially when they result in significant downtime. In my case, the boot loop could have taken hours to diagnose and fix manually. However, using this PowerShell script streamlined the process and got my VM back up and running swiftly. The script’s automated checks and balances ensure that each step is completed correctly before moving on, reducing the risk of errors.

This experience has taught me the value of having robust tools and scripts at hand to manage and mitigate such issues efficiently. Not only did it save me time, but it also minimized the disruption to my services and operations.

I hope this guide helps you manage your Azure VMs more effectively. If you have any questions or need further assistance, feel free to reach out. Happy troubleshooting!

For an in-depth guide on using a PowerShell script to fix Azure VM boot loop issues, check out our related article on TrioNxAI.