Virginia Systems Outage

I’ve written here last fall about some of the problems that the Commonwealth of Virginia is having with its computer systems. Several years ago, the state entered into a ten-year, $2.3 billion contract with Northrop-Grumman to modernize and run all the state’s computer systems and networks.  At the time, many state agencies were experiencing frequent system outages, apparently because the new systems had been designed without sufficient redundancy in their communication links.

It seems that the project has still not managed to put its problems behind it.  According to an Associated Press article carried by the Hampton Roads PilotOnline, the Virginia Information Technologies Agency [VITA] is currently trying to recover from a major system outage.  The problem apparently began at one of VITA’s data centers, outside of Richmond, with what is described as the failure of a “memory card”  (I am not sure what sort of device they mean by that).  Ostensibly, the system was designed with backup hardware and high-availability capability, but it apparently did not work too well.

The system was built with redundancies and backup storage. It was hailed as being able to suffer a failure to one part but continue uninterrupted service because standby parts or systems would take over. But when the memory card failed Wednesday, a fallback that attempted to shoulder the load began reporting multiple errors, Nixon [Sam Nixon, the state’s chief information officer] said.

The failure affected at least two dozen state agencies, including the Department of Taxation, and the Department of Motor Vehicles, which as of this morning was unable to process driver’s license application at any of its 74 offices across the  state.

The agency hoped to have most systems operational by sometime today, but said that getting all function back online might take until Monday.

I have worked on the design and implementation of highly reliable systems.   I will not claim it is always an easy job, but I know it can be done.  Mr. Nixon said that failures of the type of memory  card at fault were rare; he went on to say,

“This is supposed to be the best system you can buy, and it’s never supposed to fail, but this one did,” he said.

Whether or not it is the best system one can buy is open to question; but only an idiot thinks that there is any system that never fails — one should remember that never is a very long time.

One Response to Virginia Systems Outage

  1. […] VITA has still not recovered from the outage that started last Wednesday afternoon and brought down a significant proportion of their […]

%d bloggers like this: