Microsoft to Use Hadoop for Big Data

It is hardly a secret that, traditionally, Microsoft has not had a particularly friendly view of open-source software, having at various times likened it to communism and, according to Microsoft CEO Steve Ballmer, a cancer.  There is some evidence that Microsoft’s attitude is changing; last fall, for example, Microsoft announced that it was abandoning its Windows Live Spaces blogging platform, and would migrate its users to WordPress.com — that is, of course, the platform I use for this blog.

This past week saw another announcement about Microsoft’s adoption of an open-source technology.   As reported in an article at Wired, Microsoft has decided not only to back the provision of the open-source Hadoop big-data system on its Windows platform, integrating it with future releases of SQL Server, but also to drop development of its own system to do essentially the same thing; the announcement came in a TechNet blog post from Microsoft’s High-Performance Computing team.

Last week, a blog post from Redmond announced that the company would stop development on LINQ to HPC, aka Dryad, a distributed number-crunching platform developed in Microsoft’s Research Lab. Instead, the company will focus on its effort to port Hadoop to its Windows Server operating system and Windows Azure, its online service for building and deploying applications.

Microsoft had announced the planned availability of Hadoop back in October, but now has apparently decided to accept the platform fully.  Hadoop is designed for the analysis of very large, unstructured data sets; it got its real start at Yahoo!, and is also used by Facebook, Twitter, and eBay, among others.  The Hadoop software project is now under the stewardship of the Apache Foundation.  It was originally developed in Java for Linux, so Microsoft will have to port it to the Windows platform.  That work will eventually be fed back into the main Hadoop project.

Doug Leland, general manager of product management for SQL Server, told Wired that the company plans to eventually release its work back to the open source community.

These changes probably are something of a bitter pill for Microsoft, since they are a de facto acknowledgement that the company will not be able to control the future of personal computing as it has controlled the past.  But the changes are good news for users, most of whom care more about getting the best tools than they do about where they come from.