Data Centre Alley, a 30-square-mile stretch outside Washington DC and home to more than 200 data centres, consumes roughly the same electricity as Boston. So power company officials were alarmed when a big chunk of those centres – 60 of them – suddenly dropped off the grid one day last summer and switched to on-site generators.
The mass reaction was triggered by a standard safety mechanism across the data centre industry, intended to protect computer chips and electronic equipment from damage caused by voltage fluctuations. But it caused a huge surge in excess electricity, according to federal regulators and utility executives.
The magnitude of the imbalance forced grid operator PJM and local utility Dominion Energy to scale back output from power plants to protect grid infrastructure and avoid a worst-case scenario of cascading power outages across the region.
The near-miss – reported here in detail for the first time – forced federal regulators to recognise a new vulnerability of America’s electrical grid: unannounced disconnections by data centres.
“As these data centres get bigger and consume more energy, the grid is not designed to withstand the loss of 1,500-megawatt data centres,” John Moura, Director of Reliability Assessment and System Analysis for NERC, told Reuters in an interview. “At some level it becomes too large to withstand unless more grid resources are added.” Historically, grid operators have planned for large power plants tripping offline. But the rapid expansion of data centres processing the vast amounts of information used for AI and crypto mining is forcing grid operators to plan for new contingencies and complicating the already difficult task of balancing the country’s supply and demand of electricity.
“What it tells us is that the behaviour of data centres has the potential to cause cascading power outages for an entire region,” said Alison Silverstein, a former senior adviser to the chairman of the US Federal Energy Regulatory Commission.
The event last July 10 occurred near the DC suburb of Fairfax, Virginia, an area known as Data Centre Alley for its concentration of facilities serving Microsoft, Google and Amazon. About 70% of the world’s Internet traffic flows through the area. A month after the incident, the North American Electric Reliability Corporation (NERC), the federal regulator for grid reliability, founded a taskforce to study en masse disconnections by data centres and crypto miners.
For this story, Reuters examined thousands of pages of regulatory documents and interviewed about a dozen industry executives to determine the origins of the fault – a failed surge protector on Dominion’s Ox-Possum 230-kilovolt line near Fairfax, Virginia – and its spread across the area. NERC reviewed the incident in a report in January but did not disclose the exact location of the fault, the number of data centres involved, or how PJM and Dominion worked to rebalance the grid’s supply and demand of electricity.
The number of near-miss events like the one in Data Centre Alley has grown rapidly over the last five years as more data centres come online.
The amount of power used by data centres has tripled over the past decade and could triple again by 2028, according to a report produced by the Lawrence Berkeley National Laboratory for the Department of Energy in December. A Reuters review of disclosure filings by the Electric Reliability Council of Texas (ERCOT), the state’s main grid operator, identified more than 30 near-miss incidents since 2020, triggered by big energy users like data centres and crypto miners switching offline.
In December 2022, a failed transformer at a substation in west Texas caused nearly 400 crypto miners, data centres and oil and gas production facilities to unplug without warning.
The mass exodus produced an oversupply of nearly 1,700 megawatts of electricity – equivalent to about about 5% of the grid’s total demand – and forced 112 megawatts of power generation to shut down, according to ERCOT.
The risk of power outages will only grow as new data centres come online, the NERC forecast in a December report. Nearly all of the United States will face higher risks of energy shortfalls over the next 5 to 10 years, the report said.
The regulator urged utilities to consider updating federal reliability standards for data centres and crypto miners.
Many data centres are engineered by their operators to switch to local generators at the smallest hint of a problem on the grid to minimise the risk of an interruption to services like Google search or crypto mining, according to NERC.
Some grid operators have proposed requiring data centres to “ride through” routine voltage dips without disconnecting. But data centre operators are opposed because of the risk of damaging electronic equipment and cooling systems.
ERCOT last year withdrew a proposal that would have imposed ride-through restrictions on data centres and crypto miners after facing pushback from an industry group, the Data Centre Coalition.
The group, whose members include Amazon, Google, and Meta, cited costs and the risk of damaging computer chips and cooling systems exposed to fluctuating voltage levels.
“Data centre hardware and power supplies, similar to other electronics, are very sensitive to power supply stability,” the coalition said in January 2024 comments filed with ERCOT.
“Deviating from this range will deteriorate the optimal performance, reduce longevity, or damage the components beyond repair.”
The coalition said in a statement to Reuters that it intended to be a helpful partner to grid operators.
“We fully recognise grid planning and management is the responsibility of utilities and grid operators, but DCC is committed to leaning in as an active and engaged partner to be helpful and ensure we collectively meet this moment,” said Aaron Tinjum, the coalition’s Vice-President of Energy.
Amazon, Google and Meta did not return messages seeking comment. ERCOT did not return messages seeking comment. There is “high potential” for the magnitude of these disconnection events to grow as larger operations plug into the Texas grid, ERCOT operations engineer Patrick Gravois said in a December presentation to NERC’s Large Load Task Force.
Gravois said the grid operator is still working to determine exactly what prompts big users of electricity to unplug from the grid, so that it can avoid surprises.
Ari Peskoe, director of the Electricity Law Initiative at Harvard Law School, said regulators could require data centres to ride through voltage dips – but that could risk Big Tech decamping for states with more relaxed rules.
Jim Simonelli, chief technology officer for Schneider Electric’s secure power division, said utilities and the data centre industry have a lot of lessons to be learned from what happened outside Washington DC this past July.
“One thing that doesn’t exist yet for the data centre industry is how to be grid-friendly,” Simonelli said. - Reuters