August 15, 2006

http://www.hpcwire.com/hpc/810809.html

It is no secret in the HPC community that Linux clusters are becoming increasingly popular with market share growing significantly over the last two years. This growth is only sustainable if Linux clusters continue to move into new areas of growth and unlock new market segments.

The growing appeal of using Linux clusters in organizations often boils down to choice and the freedom to select and build a custom solution that meets individual needs. However, as Linux grows in popularity the element of choice is inhibiting broader market adoption because of the lack of stability and consistency that is inherent.

Linux clusters remain complex, decoupled and problematic. This current state is impeding growth in areas at the low-end or mass market that benefit from Linux clusters, who either do not have the knowledge to build and configure a cluster on their own or those that simply do not have the desire or interest in doing so.

Early adopters are willing to go through the pain of putting together systems to gain some competitive advantage, but for the majority of the market, this isn't a viable approach. In fact, many IT departments have a mandate to buy a solution and not build it.

At the same time, inconsistent software development and certification for Linux clusters is another challenge that is inhibiting wide-scale adoption of Linux clusters. For example, a software developer who has created a business application, has the responsibility to deliver a consistent customer experience. With the large number of different options available, ISVs are tasked with engineering their application in such a way that it functions as designed on a wide range of hardware and software solutions. This complexity affects not only the costs of getting a product to market, but also the costs of supporting a product once customers have deployed it.

If a customer has a custom Linux cluster set-up and their application does not work in the expected manner, the ISV is held accountable and subsequently tasked with remedying the problem. Multiply this by the matrix of different combinations of tools and devices available as part of Linux clusters and the challenge facing today's software developers is overwhelming.

In the days when UNIX servers were clustered together with systems management software provided by the hardware vendor, ISVs and consumers alike were sheltered from the fallout of choice. However, employing both proprietary hardware and software meant that solutions were expensive, closed and from a single source.

Today, the effects of commoditization are visible across the solution stack up to and including the middleware. The outcome, a set of decoupled software components, give users open solutions from a wide variety of vendors at a low cost. Moving forward, there is an opportunity to maintain these characteristics but also provide superior value and best of breed functionality through a standards based solution.

Customers can now buy bundled, pre-integrated clusters from a growing number of vendors such as Dell at attractive price points. They typically combine industry standard and commodity hardware components, featuring best of breed, low cost open source software stacks. Currently, vendors differentiate themselves from one another based on the whole product solution and the support and services it provides for these clusters. In the past, differentiation was delivered through engineering prowess and technology advancement.

With an end-to-end solution based on standards available from hardware vendors, customers would have the opportunity to benefit from high performance computing by quickly deploying an out of the box product. In essence this would enable a new class of users to apply 'supercomputing' technologies against a wave of applications. This would serve to dramatically grow the market for HPC clusters and ultimately virtualization clusters. In fact, it can be imagined that a single software stack could support the majority of applications in use today.

Traditional HPC customers typically have built custom, complex environments, which require flexible solutions to meet their needs. As mentioned earlier, there is a larger growth opportunity at the low-end where customers value simplicity, availability and consistency and are currently sacrificing these three things as they choose their scale-out Linux implementations.

By creating a product that matches the buying patterns of end users and establishing a consistent application development environment, the door begins to open into these new areas. Specifically, a standardized cluster software stack implemented across the industry gives end users a basic cluster solution that is stable and ready to go.

To the customer, the benefits of standards will clearly benefit the bottom line. Standards enable a lower total systems price by reducing the operating cost of scale-out systems as vendors adopt common components and invest in usability and consistency rather than engineering advancements in hardware and software layers that are, frankly, "good enough."

ISVs and hardware vendors benefit from significantly simplified product distribution because of the reduced complexity and costs of bringing product to market. Customers then reap capital cost savings while benefiting from new innovations faster. By significantly reducing capital and operating costs, IT departments will operate cost effectively and can not only maintain service levels but also likely increase them over time.

Software developers will be able to certify their applications on a consistent Linux cluster platform which will reduce the support burden, leaving valuable development resources free to innovate on the platform. In addition, as more customers adopt the architecture in support of a wider range of business problems, ISVs will see increased returns from their investments, taking Linux clusters one step closer to a broader market. The ability to access growing marketing segments, such as, manufacturing, energy and financial services where these applications are increasingly being deployed on scale-out architectures will unlock new opportunities for software companies who are looking to expand their market reach.

Ultimately, grid technology will be just that, a set of technologies that are a part of a bigger product and not a point product itself. A standardized cluster software stack will give customers running applications as diverse as billing to ERP an optimized infrastructure to perform these tasks and realize the benefits and potential of what a Linux cluster has to offer.

There is a clear framework that needs to be followed in order to access and accelerate the market. Quite simply, customers want pre-integrated, packaged solutions that are easy to deploy. This package needs to be easy to buy and meet a defined level of performance expectation by delivering a standard of consistency that encourages development of relevant software applications.

Naturally for this concept to succeed, it requires the support of the industry players in both hardware and software as well as industry and standards groups, such as the OGF. By providing a viable alternative to a scale-up infrastructure, a standardized cluster software stack will take cluster computing to the next level and continue the growth and momentum that we have seen over the past few years into new territories.

Gary Tyreman will be kicking off the LinuxWorld HPC track on August 15th at 10:15am with his panel, "HPC 101: Building Blocks & Standards."