At 6:00 a.m. on April 13, 2023, the IT team at the University of Health Sciences and Pharmacy got a call: “VPN’s down, AD is acting up.” The CIO tried to log in remotely. Nothing. By 6:15, the data center looked “dark” — no network reachability at all. What looked like a routine outage would become an 11-day ordeal involving four dead hypervisor hosts, a premature “all clear” announcement, an opportunistic phishing wave, and a second, more damaging crash of the exact same hardware.
The organization eventually recovered. But the post-mortem — prepared by Zachary Lewis, CIO & CISO at the University — reads less like an apology and more like a gift: a candid, detailed account of what broke, what the team got wrong, and 28 concrete actions any security or IT leader can put to work before their own “bad day” tries to become a crisis.
The Root Cause Was Boring. That’s the Point.
There was no nation-state actor. No zero-day. No ransomware note. The root cause was a RAID controller hardware failure that corrupted the boot partitions on all four of the organization’s ESXi hypervisor hosts — taking every virtual machine on them offline at once: Active Directory, Exchange, the file server, and the building management system.
It’s a useful reminder that “crisis” doesn’t require a sophisticated adversary. Aging infrastructure, missing monitoring, and slow escalation can do the job just as well — and CISOs who fixate only on threat actors risk under-investing in the unglamorous plumbing that keeps everything running.
How the Week Actually Unfolded
The timeline is worth sitting with, because it shows how each small gap compounded into the next:
- 06:00–07:10: A physical inspection finally finds the smoking gun — a dangling cable and hosts that have silently rebooted. Nobody had automated alerting that would have caught this sooner.
- 08:00–12:00: All four ESXi hosts are confirmed to have corrupted boot partitions. Every VM is down. The DR plan is invoked, but there’s no runbook written specifically for a hypervisor failure of this kind.
- Overnight: The team discovers there are no hot-spare servers pre-staged. Replacement parts have to be sourced on the fly.
- 14-Apr, evening: With services only partially validated, leadership holds a town hall and declares systems “fully restored.” No formal verification had been completed — the announcement is based on assumption, not evidence.
- 15-Apr: A “75 GB of data stolen” email starts circulating. EDR shows no indicators of compromise; it’s almost certainly opportunistic phishing riding on the outage’s publicity — but it adds fear and noise at the worst possible time.
- 20-Apr: The same hosts crash again. This time the incident is escalated to leadership and the full IR playbook — contain, eradicate, recover — is finally activated. Forensics trace it back to the same failing RAID controllers, never fully remediated the first time.
- 21–23 Apr: A fresh wave of phishing emails, this time explicitly referencing the outage, floods inboxes. The organization responds with an MFA push, privileged password resets, and a mass EDR scan.
- 24-Apr: Eleven days after the first phone call, services are finally stable and the post-mortem begins.
Five Gaps That Turned an Outage into a Crisis
Strip away the specifics and the case boils down to five systemic weaknesses — each one common enough that most organizations will recognize at least one:
- No automated detection of hypervisor failure. The team found out from a physical inspection, not a dashboard.
- Incident response wasn’t escalated to “critical” fast enough. Classification depended on judgment calls rather than a predefined trigger.
- DR runbooks had no hypervisor-failure scenario. The plan existed; it just didn’t cover the failure mode that actually happened.
- Public statements went out before verification. The “fully restored” announcement created false confidence — for leadership and for attackers watching the outage unfold.
- Hardware was past its recommended refresh window. A known, manageable risk that had gone unaddressed.
Notice that none of these are exotic. They’re the kind of gaps that show up in a tabletop exercise or a maturity assessment — if anyone runs one.
What Every Organization Should Actually Do
The post-mortem’s real value is its action list — 28 items, each tagged with a rationale and priority, organized by function. A few of the highest-leverage ones:
Detection & Monitoring
- High priority: Deploy hypervisor health monitoring (vSphere/vCenter alarms, Prometheus, or equivalent) feeding boot-partition and storage health into the SIEM.
- High priority: Build a correlation rule that fires a “Critical Service Loss” alert the moment AD, VPN, and Exchange go unreachable simultaneously.
Incident Response & Escalation
- High priority: Adopt a risk-based triage matrix: loss of authentication, VPN, or email is an automatic “Critical” classification, removing subjectivity from the first call.
- High priority: Require four-eye approval (IR Lead + Comms Lead) before any external communication goes out — directly addressing the premature “fully restored” statement.
- High priority: Maintain a specific runbook for hypervisor/boot-partition failure, including hardware swap steps and firmware checklists.
Disaster Recovery & Business Continuity
- High priority: Implement hybrid identity — a synchronized, tested cloud AD (Azure AD or Okta) as failover for on-prem AD.
- Medium priority: Stand up a warm-spare pool of identical hardware, including RAID controllers, stored off-site and ready to deploy.
Hardware Lifecycle
- High priority: Audit inventory and flag anything over five years old for replacement.
- Medium priority: Deploy predictive maintenance tooling that tracks SMART data and RAID controller health before it becomes a 6 a.m. phone call.
Identity & Endpoint Security
- High priority: Enforce MFA on all privileged accounts organization-wide.
- High priority: Integrate real-time sandboxing for inbound attachments referencing recent incident keywords (“outage,” “data stolen”) — the exact vector opportunistic phishers used here.
Communications
- High priority: Pre-approve status-update templates that require IR verification before release.
- Medium priority: Assign a single point of contact for executive updates, backed by a live incident dashboard.
Governance
- Medium priority: Track Mean Time to Detect and Mean Time to Resolve for every incident, and set improvement targets against them.
- High priority: Run a post-mortem meeting with technical, legal, PR, and executive stakeholders in the room, with actions captured in a tracker.
A Realistic Rollout, Not a Wish List
Twenty-eight action items can look overwhelming next to a Tuesday full of tickets. The post-mortem’s implementation roadmap breaks it into something achievable:
| Timeframe | Focus | What Happens |
| Weeks 1–2 | Detection & escalation | Hypervisor alarms, critical-service correlation rule, four-eye comms approval |
| Weeks 3–4 | DR & runbooks | Hypervisor-failure runbook, Azure AD failover test, warm-spare stash |
| Weeks 5–8 | Hardware & vendors | Inventory audit, tag aging servers, negotiate critical-incident SLAs |
| Weeks 9–12 | IAM & endpoint hardening | MFA rollout, post-incident password rotation, attachment sandboxing |
| Months 4–6 | Exercises & improvement | Quarterly hypervisor-outage tabletop, IR maturity check-in, post-mortem published |
| Ongoing | Governance & metrics | MTTD/MTTR tracking, knowledge base upkeep, annual contract and DR review |
The Real Lesson: Verify Before You Reassure
If there’s one line from this incident that should stick with every CISO, it’s the gap between the 14-Apr town hall — “systems fully restored” — and the 20-Apr repeat failure that proved otherwise. Confident communication is a leadership skill. Confident communication without verification is a liability, and in this case it directly undermined the organization’s credibility six days later.
Resilience isn’t built in the moment of crisis — it’s built in the weeks beforehand, in the alarms that get configured, the runbooks that get written for the failure mode nobody’s thought of yet, and the discipline to say “we’re still validating” instead of “we’re done.” A single RAID controller took this organization offline for eleven days. The fix wasn’t exotic technology — it was monitoring, a triage matrix, a tested failover, and the willingness to verify before reassuring.
Zachary Lewis
Zach Lewis currently serves as the CIO and CISO for the University of Health Sciences and Pharmacy and chairs the Enterprise Risk Management committee. Zach has over 13 years’ experience working in Tech Start-ups, Energy, Higher Education, and Consulting. He is an officer in the St. Louis Chapter of InfraGard, which is a partnership between the FBI and private sector for the protection of U.S. Critical Infrastructure.
LinkedIn: https://www.linkedin.com/in/zacharylewis1/
On Amazon: https://www.amazon.com/dp/1394357044
