Friday morning brought BSODs on Windows endpoints running Crowdstrike Falcon, snowballing into massive outages and interruption of services around the world. See below for steps to effect a workaround.
Summary
Crowdstrike’s statement summarizing the situation:
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.
The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”
Details
- Symptoms include hosts experiencing a bugcheckblue screen error related to the Falcon Sensor.
- Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
- Windows hosts which are brought online after 0527 UTC will also not be impacted
- Hosts running Windows 7/2008 R2 are not impacted
- This issue is not impacting Mac- or Linux-based hosts
- Channel file “C-00000291*.sys” with timestamp of 0527 UTC or later is the reverted (good) version.
- Channel file “C-00000291*.sys” with timestamp of 0409 UTC is the problematic version.
What To Do?
Crowdstrike has released an update to fix the issue, but if endpoints are crashed, there is a workaround to bring them back online:
Workaround steps for public cloud or similar environment including virtual
Option 1:
- Detach the operating system disk volume from the impacted virtual server
- Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
- Attach/mount the volume to to a new virtual server
- Navigate to the %WINDIR% |System32drivers CrowdStrike directory
- Locate the file matching “C-00000291* sys*, and delete it.
- Detach the volume from the new virtual server
- Reattach the fixed volume to the impacted virtual server
Option 2:
- Roll back to a snapshot before 0409 UTC.
Need Help?
If the Crowdstrike failed update has left your organization in a tough spot, reach out to our parent company Novacoast—an industry-leading MSSP and IT Services provider with decades of experience in incident response.
Finding Affected Assets
CrowdStrike has also released a query to help identify impacted assets:
// Get ConfigStateUpdate and SensorHeartbeat events
#event_simpleName=/^(ConfigStateUpdate|SensorHeartbeat)$/ event_platform=Win
// Narrow search to Channel File 291 and extract version number; accept all SensorHeartbeat events within impact window
| case{
#event_simpleName=ConfigStateUpdate | regex("|1,123,(?.*?)|", field=ConfigStateData, strict=false) | parseInt(CFVersion, radix=16);
#event_simpleName=SensorHeartbeat | rename([[@timestamp, LastSeen]]);
}
| case{
#event_simpleName=ConfigStateUpdate | @timestamp>1721362140000 AND @timestamp < 1721366820000 | CSUcounter:=1;
#event_simpleName=SensorHeartbeat | LastSeen>1721362140000 AND LastSeen<1721366820000 | SHBcounter:=1;
*;
}
| default(value="0", field=[CSUcounter, SHBcounter])
// Make sure both ConfigState update and SensorHeartbeat have happened
| selfJoinFilter(field=[cid, aid, ComputerName], where=[{ConfigStateUpdate}, {SensorHeartbeat}])
// Aggregate results
| groupBy([cid, aid], function=([{selectFromMax(field="@timestamp", include=[CFVersion])}, {selectFromMax(field="@timestamp", include=[@timestamp]) | rename(field="@timestamp", as="LastSeen")}, max(CSUcounter, as=CSUcounter), max(SHBcounter, as=SHBcounter)]), limit=max)
// Perform check on selfJoinFilter
| CFVersion=* LastSeen=*
// Calculate time between last seen and now
| LastSeenDelta:=now()-LastSeen
// Optional threshold; 3600000 is one hour
| LastSeenDelta>3600000
// Calculate duration between last seen and now
| LastSeenDelta:=formatDuration("LastSeenDelta", precision=2)
// Convert LastSeen time to human-readable format
| LastSeen:=formatTime(format="%F %T", field="LastSeen")
// Enrich aggregation with aid_master details
| aid=~match(file="aid_master_main.csv", column=[aid])
| aid=~match(file="aid_master_details.csv", column=[aid], include=[FalconGroupingTags, SensorGroupingTags])
// Convert FirstSeen time to human-readable format
| FirstSeen:=formatTime(format="%F %T", field="FirstSeen")
// Move ProductType to human-readable format and add formatting
| $falcon/helper:enrich(field=ProductType)
| drop([Time])
| default(value="-", field=[MachineDomain, OU, SiteName, FalconGroupingTags, SensorGroupingTags], replaceEmpty=true)
| case{
CSUcounter=0 AND SHBcounter=0 | Details:="OK: Endpoint did not receive channel file during impacted window. Endpoint was offline.";
CSUcounter=0 AND SHBcounter=1 | Details:="OK: Endpoint did not receive channel file during impacted window. Endpoint was online.";
CSUcounter=1 AND SHBcounter=1 | Details:="CHECK: Endpoint received channel file during impacted window. Endpoint was online. Endpoint has not been seen online in past hour.";
}
Sources
- Crowdstrike Advisory
https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/