T-SQL Tuesday #202: The Saturday SQL Slammer Took Both Our Data Centers

This month Marlon Ribunal is hosting T-SQL Tuesday #202, and he's asking about that one SQL Server outage you'll never forget. I've had a few, but this one beats them all.

Saturday, January 25th, 2003. My cell phone rang at about 3AM. It was Microsoft calling to tell me what was happening inside my servers.

We had VIP support at the time, which is why I heard it from Redmond before I heard it from my own building. I called our head of IT expecting to wake him -- but he was already on site with his Team, trying to work out what the hell was going on.

What Was Actually Happening

SQL Slammer. First detected on the internet at 05:30 GMT that morning, which put my phone call roughly about three hours into it.

It was a single 404-byte UDP packet aimed at port 1434, and it sent so many copies of itself that servers and routers buckled and network traffic slowed to a crawl. No payload, no user interaction. Just pure replication exploiting a buffer overflow in Microsoft SQL Server 2000 and MSDE database products. Because the worm lived entirely in memory, a restart wiped it, but an unpatched reboot invited it right back in.

The sheer math made it the fastest-spreading worm in history. It doubled its infected host count every 8.5 seconds, tearing through 75,000 victims in under ten minutes.

The tragedy? The patch had been live since July 24, 2002 (MS02-039). It had already been sitting on the shelf for six months on the morning the world woke up to it.

How It Got In

Our firewall held. Port 1434 was not open to the internet and nothing came through the perimeter.

It came in over the VPN. One of our senior developers had a connection open, doing legitimate work, the way he did most weekends. That connection was authenticated, sanctioned, and exactly what we had built the VPN to allow.

Once it was inside, there was nothing between hosts to slow it down. It started with one or two SQL Servers and then it had the entire network.

The Routers Went First

The sheer volume of UDP scan traffic was overwhelming, and the routers went down under it. Not degraded -- our network lead's words were that they had halted to a stop. We couldn't even log in to manage them.

That's a particular kind of bad. The equipment you would use to contain a network problem is on the far side of the network problem. Everything we would normally reach for was unreachable.

From where we sat, the systems went down one by one as the internal network collapsed around them -- until both sites were completely down.

Both Sites

Chicago was primary and New York was the DR site. Financial services, so Monday's market open was not a time that could be negotiated.

We lost both sites.

That is the part I have never forgotten. Geographic redundancy is built for a site going away -- fire, flood, power, a backhoe through a fiber run. It is not built for something that gets into the network and reaches both your primary and DR sites in the same instant. New York wasn't a fallback that weekend. It was a second casualty.

The Recovery

We shut everything down. The IT Team took the routers offline and disconnected the edge switches and distribution routers from the core, to give us a network quiet enough to work in.

Then we recovered from the servers outward. One at a time. More than 400 physical servers across the two sites.

There was no clever fix available. The worm was memory-resident, so the machine came back clean, and it would be reinfected in seconds if we put it back on a network that still had infected hosts. The only way through was to do them in order, keep the clean ones isolated from the dirty ones, and not lose track of which was which at any point during recovery.

The IT Team and the DBA Team worked it together, Saturday into Sunday into Monday. Full recovery by about 4AM Monday morning. We opened when the market did and nobody outside the building knew.

Getting the Servers Back Was the Easy Half

Recovering 400+ servers was the part with a clear finish line. You know when an operating system is back online. The harder problem was everything that had to be true before we could begin trading at Monday's open.

In many cases the state was simply gone. Trader permissions and account data had to be repopulated from backups so that we could account for open orders -- and open orders don't close just because your data center did.

Then came reconciliation with the exchanges, verifying current state before we could open clean. That is not just a database task. It is a full business process, and it had to be done end-to-end before recovery was complete.

The Ashtray

Sometime during that weekend the owner of our company came down to L2 and handed me an ashtray, so I could smoke while working the recovery. Twenty-three years later, that is still the detail I remember first.

What I Took From It

The patch had existed for six months. That is the sentence everyone quotes about Slammer, and it is true, but it was never the interesting part to me.

The interesting part is that our perimeter did its job and it did not matter. Port 1434 was closed to the internet, and the worm still came in over an approved VPN connection opened by one of our most senior developers doing legitimate work on a Saturday. Nobody did anything wrong.

That still happens, and it does not need a worm. I wrote about a SQL Server CVE on Wednesday that lives in SSMS rather than in the database engine. The vulnerable machine is the workstation somebody administers from, and your SQL Server patch inventory never looks at it -- for the same reason a developer's laptop wasn't on anyone's list in 2003.

More to Read

T-SQL Tuesday #202 Invitation -- That One SQL Server Outage You'll Never Forget
MSFT Security Bulletin MS02-039
F-Secure -- Worm:W32/Slammer
SQL Slammer
sqlfingers -- This SQL Server CVE Isn't on Your SQL Server

This SQL Server CVE Isn't on Your SQL Server

I'm in Italy right now, which is why I've been quiet for a couple of weeks. MSFT picked yesterday to ship the biggest Patch Tuesday on record. SecurityWeek counts 974 CVEs, and 62 of them land in Microsoft's SQL bucket. That's a lot of patching to sort through, but one of them caught my attention for a completely different reason.

CVE-2026-65669 has a CVSS score of 9.6, and MSFT titles it Microsoft SQL Server Elevation of Privilege Vulnerability. Reading that title, I would naturally expect to be looking for a vulnerable SQL Server build -- but I wouldn't be. The vulnerability is in SQL Copilot inside SSMS 22, which means the affected software could be sitting on a DBA's workstation rather than on a SQL Server at all.

That's where this one gets interesting.

Your SQL Server Can Be Fully Patched

Every step of my normal SQL Server patch process starts with a server. I pull the build, check the servicing path, determine what applies, prioritize it, schedule the work and eventually verify that the instance was patched. A vulnerability scanner working against the server estate is doing essentially the same thing from a different direction.

CVE-2026-65669 can completely escape that process because the affected product isn't necessarily installed on any of the servers I'm checking. The SQL Server itself can be current, the vulnerability scan can report it clean, and both statements can be accurate while a vulnerable copy of SSMS is sitting on a completely separate workstation used to administer the SQL Server(s).

The compliance report isn't wrong. It answered the questions we asked -- we just didn't ask about the machines running SSMS.

The Permissions Are the Bigger Story

According to Automox's analysis, an attacker can convince a user to submit specially crafted instructions to SQL Copilot, allowing the assistant to bypass its intended read-only restrictions and perform database activity using the permissions of the account already connected through SSMS. ZDI describes the same basic chain: get the user to submit specially crafted instructions to SQL Copilot, and the resulting database access occurs at that user's permission level.

That last part matters more to me than the 9.6 score. This vulnerability doesn't grant permissions. It uses the ones already on the connection. If somebody opens SSMS against production as db_owner, that's the authority sitting behind the session. If they connect with an account limited to the SELECT permissions actually required for their work, the potential impact is very different.

We've been talking about least privilege since long before anybody put Copilot in Management Studio. The difference now is that the person at the keyboard is no longer the only factor we have to consider when deciding how much authority an interactive connection should have.

The Patch Doesn't Fix the Permission Problem

Obviously, the affected copies of SSMS need to be updated. That's the immediate remediation for this vulnerability. What the patch doesn't change is the permission model behind those connections.

If DBAs and developers routinely connect to production with write permissions when they're only there to query data, those permissions will still exist after SSMS is patched. The CVE didn't create that exposure; it just gave us a very good reason to look at it more closely.

Find Out Who Is Connecting Through SSMS

You can't inventory the SSMS version installed on somebody's workstation from T-SQL, but SQL Server can tell you about the Management Studio sessions currently connected to it. Run this during normal working hours, because sys.dm_exec_sessions only tells you what is connected now:

SELECT
    s.login_name,
    s.program_name,
    s.host_name,
    COUNT_BIG(*) SessionCount,
    MIN(s.login_time) EarliestLogin
FROM sys.dm_exec_sessions s
WHERE s.is_user_process = 1
AND s.program_name LIKE N'%Management Studio%'
GROUP BY
    s.login_name,
    s.program_name,
    s.host_name
ORDER BY
    s.login_name,
    s.host_name;
GO

There is an important caveat here. program_name is supplied by the client, so this is useful inventory information, not something I would ever treat as a security control. What I want from the query is much simpler -- who is routinely pointing Management Studio at this SQL Server, and from which machines?

Those machines are now part of my SQL patching inventory.

Then Look at What Those Accounts Can Do

This is where I think the CVE becomes more useful than simply another item on the September patch list. Once I know which identities are being used for interactive SSMS connections to production, I want to know what authority those identities actually have.

And I don't mean just looking for db_owner. Effective access can come from server roles, database roles, direct grants, custom roles, Windows group membership and ownership. The useful question isn't whether a login belongs to one particular dangerous role. It's what can this identity actually do once it connects?

Don't Forget the Jump Box

The first place I'd look is the shared administrative server or jump box, if you have one. Those machines tend to sit in an awkward spot operationally. They're not SQL Servers, so the SQL Server patch inventory may not include them, and they're not normal end-user workstations either.

At the same time, they're often exactly where SSMS is installed, where production connections are saved, and where some of the most privileged database work in the environment takes place. If I'm looking for SSMS 22 installations that matter, I'm starting there.

What I'm Taking Away From This One

There are a lot of SQL-related CVEs to deal with this month, and the applicable SQL Server patches obviously still need to be evaluated, tested and deployed. But CVE-2026-65669 exposed something different in the way I normally think about SQL Server vulnerability management.

My process starts with the SQL Server. This vulnerability doesn't.

I could inventory every SQL Server, verify every build, install every applicable update and produce a completely accurate compliance report without ever examining the machine containing the vulnerable software. That's not really a patching failure. It's a scope failure.

So I'm adding another question to the process -- what tools are being used to administer SQL Server, where are they installed, and what permissions sit behind those connections?

Patch the SQL Servers and patch SSMS. Then take a look at the accounts people are using when they open SSMS against production. The CVE will eventually disappear from the vulnerability report. Those permissions won't.

More to Read

MSRC -- CVE-2026-65669
ZDI -- The September 2026 Security Update Review
Automox -- Patch Tuesday September 2026 CVE Analysis
SecurityWeek -- MSFT Patches Record 974 Vulnerabilities