Winds Wallop the Cloud

July 4, 2012

Most US readers have probably seen reports in the news media of the huge storm that affected the US, from the mid-west to the mid-Atlantic states, beginning Friday afternoon, June 29, and extending into the early hours of Saturday, June 30.  The storm, a line of severe thunderstorms plus a windstorm, a type known as a derecho, traveled generally eastward at speeds of about 85 km/hr (60 mph), and packed wind gusts in excess of 70 mph when it reached us here in the Washington DC metro area.  I can personally testify that it was impressively noisy.

We were very fortunate that, in our immediate area of Northern Virginia, west of Washington DC, we had only minimal storm damage, and have not (touch wood!) had any significant power outages.  That last is very good news when daytime temperatures of 100 F (38 C) make air conditioning close to a necessity.  Many other folks nearby were not so lucky.  As always with thunderstorms, though, the effects varied widely over a relatively small geographic area.  Amazon has a data center a handful of miles from here, which lost power during the storm, and, owing to a chain of problems, was offline for several hours, affecting some of Amazon’s “cloud” customers, including Instagram and Netflix.    Many people in the affected areas are still, four days later, without electricity.

The folks over at the SANS Internet Storm Center have posted a diary entry with an anonymous report from a sysadmin at an affected data center.  The story is, I think, well worth reading, because, at least in my experience, nothing in it sounds very much out of the ordinary for this kind of event.

The story that follows offers some lessons relearned and possibly a few new ones.

It’s only a few paragraphs long, so I urge you to have a read; I would emphasize these points:

  • Mother Nature is not obliged to follow the script in your emergency/disaster plans.
  • Having good solutions to the technical problems posed by a disaster is a Very Good Thing.   However, low- or no-tech problems can be as bad or worse (e.g., how do you get key people on site if there are problems with the roads?).
  • Emergency recovery procedures and data (e.g., critical configuration parameters and passwords) need to be documented and accessible, even when systems are not available.  (Paper still has its uses.)

One of the things that makes planning for this kind of event difficult is the habits of mind we acquire, largely unconsciously.  Under ordinary circumstances, for example, electrical power from the grid is just there.  Since I can drive from my house to the data center in ten minutes, 30 minutes of UPS power is ample.  In a way, this problem is like debugging, or like security evaluation.  Commonly, in looking at potential solutions to a problem, we ask, “Will this work?”.   With debugging, security evaluation, or disaster recovery planning, the question has to be something like, “How can this possibly go wrong?”.

I’m grateful to those, like the anonymous reporter to the SANS ISC, who share their experiences with us.


Another Look at DC Power

April 25, 2012

Back in December, I posted a note about the resurgence of interest in DC power distribution systems, especially within data centers.  Although large scale electricity distribution systems (such as regional or national grids) have used AC for years, since the resolution of the “War of the Currents“,and obviously constitute a workable solution — I am, after all, writing this at about 10:00 PM — the data center environment differs in some significant ways from that of the average utility customer.  The electronic devices themselves almost all work on DC power  (converted from the AC supplied by the grid; backup power supplies for emergencies almost always use batteries, which supply DC.   As I noted in that earlier post, the use of DC distribution in large data centers could potentially produce significant increases in energy efficiency.

Technology Review has a new article that discusses the possible use of DC power distribution on a larger scale.   According to Greg Reed, director of the Power & Energy Initiative at the University of Pittsburgh, the growth in the use of electronic devices, especially consumer electronic devices, has meant that a larger amount of the total demand for power is, ultimately, for DC.  Currently, this DC power is supplied by the battery chargers, power supplies, and “wall warts” of our PCs, smart phones, flat-screen TVs, and other gadgets.  Reed thinks that this trend will continue.

“Within the next 20 years we could definitely see as much as 50 percent of our total loads be made up of DC consumption,” he [Reed] says. “It’s accelerating even more than we’d expected.”

He goes on to argue that a “DC takeover” of the grid is “inevitable”, due to improvements in efficiency, from eliminating AC/DC conversions, and to the increased use of consumer electronics, solar panels, and LED lighting, all of which are more “at home” in a DC-powered world.

It is certainly true that there is technology today, unavailable in Edison’s time, that makes high-voltage DC transmission over significant distances feasible.  I expect this type of distribution will be used more in the future for installations where it makes sense.  I also think that the use of DC power distribution in data centers will increase; moreover, this kind of local grid installation probably makes sense in a number of other contexts, like large commercial buildings.  Electric vehicles, too, use batteries that are recharged with DC power, so there is probably a role for local DC grids there.

Dragan Maksimovic, an expert in power electronics at the University of Colorado in Boulder, estimates that solar-powered vehicle chargers his group is developing should cut power losses from 10 percent of what the panels produce to just 2 percent.

So, I think there is a pretty good case for deployment of DC power distribution on the local scale.  In a data center, it makes little sense to have a large number of servers, each with its own power supply, taking AC from the local utility and turning it into, say, 24 volt DC.  However, I very much doubt that we will see any wholesale switch to DC power distribution on a large scale.  The US power grid represents a huge capital investment, and it does work.


%d bloggers like this: