Cluster Lifecycle Management: Recycle and Rebirth

In the previous Cluster Lifecycle Management column, I examined the various options for Capacity Planning and Reporting that can help you prepare for the inevitability of upgrading or refreshing your HPC cluster. In this column, we wrap up the Cluster Lifecycle Management series by discussing how you should actually go about upgrading or perhaps even replacing the system.
I focused quite a bit in this column series on the steps that need to be taken to keep your system running efficiently so that end users have productive experiences and your organization gains a positive ROI on its investment in HPC technology. But even with your clusters running at peak efficiency, the day will come – could be within a few months or could be four years – when you will start seeing the need to upgrade the system.
In the Capacity Planning and Reporting column, I stressed the need for a solid reporting system that keeps the HPC manager up to date regarding how the clusters are being used, and how they are performing. This is important because a system refresh, whether big or small, can’t be a snap decision. Budgeting for and procuring the new resources require a long lead time and must follow a thorough examination of how the system has been used to date. Having the usage and performance history is critical to making good decisions regarding upgrades and changes.
Symptoms that indicate change is needed will come in various forms and severities. Most commonly, your business has grown and users are asking for more speed, or more throughput, or additional storage to complete their jobs. Or perhaps the software applications have evolved and need additional compute capacity or even a different software stack to run effectively. After a couple of years, hardware can simply get out of date and can’t keep up with required throughput or can become significantly more expensive to operate and maintain. Associated symptoms include more frequent hardware failures, more unexplained intermittent job failures, etc.
Having the system and job usage and performance reporting system, in combination with a trouble-ticketing and change management system, will provide you the data you need to make data driven analytical decisions as to which options are the best ones to take. As always, very seldom is there only one path!
System upgrades and refreshes come in varying degrees. Just like making a decision on buying a car – do you replace the old one, or just upgrade/repair some parts, or buy a new one and keep the old one – cluster refreshes are no different. However, the decision is not to be taken lightly due to the cost involved with each of the alternatives. Generally speaking, the options are to upgrade existing system, or to expand the existing cluster by adding additional compute and/or storage, or to replace the entire system. In my experience with scores of customers or upgrades, rolling upgrades seem the most common where customers are able to keep certain production volume going while upgrading some parts of the system.
To determine which path to take, you’ll need to refer to the historical reporting information I recommended you maintain from the start of operations. Knowing what hardware to upgrade or replace will be dictated by the metrics of how your system has been used. These metrics will help you make a decision that provides the best ROI, as was your original decision to deploy HPC technology in the first place.
Some important information, or metrics, that you’ll have to examine include the following:

• What is the average throughput of your system?
• What is the usage profile?
• What applications run most efficiently on which architectures?
• Which nodes are used the most? The least?
• What problems are your users reporting?
• What are the costs associated with maintenance, administration, power, and software licenses?

Your job is to perform a cost-benefit analysis using these data points so you can determine which option will yield the best return to the organization. And most importantly, your enhanced system should serve the end users more effectively and reliably than ever before. Just as you did before purchasing your first cluster, you must assess your upgrade needs based on your current and expected usage profile.
Once the decision has been made on what to replace or acquire, you must crank up the RFP (Request For Proposal) process that you used to buy the original system. If possible, put the same team together – system administrators, datacenter operators, end users, IT staff, and executives, and engage them in the analysis and acquisition process again. If you weren’t happy with the architecture or performance of your original system, you have the opportunity now to enact changes that will improve your cluster in its second incarnation. One major consideration with a new system RFP is to ensure compatibility of the new system with existing one is taken into account. This includes management systems, schedulers, software and your workflows and policies and procedures.
Even if you had a positive experience with your original HPC purchase and want to buy from the same vendor again, I still recommend sending at least two or three other RFPs out. In today’s rapidly changing technological landscape, you might be surprised to find a different vendor now offers exactly what you need at a better value. Notice I said value not price! A good vendor with knowledge of your existing system can provide valuable insight into what options may or may not be a good choice.
Just as your original RFP took into account deployment of the new system, your upgrade must be installed, deployed and validated by a seasoned professional. Upgrades, especially major, require a unique skill set because they must be accomplished with minimal disruption to the existing system. If the old system will be replaced entirely, logistics can become a problem and must be handled in the RFP. All of the implementation activities add ‘soft’ dollars to the cost of the upgrade and must be taken into account in the cost-benefits analysis.
Finally, what will you do with the old hardware or entire system you are replacing? Depending on the age and condition, you might want to consider donating to a local college or even selling the system to another organization.
Once the upgrade or replacement is up and running, the rebirth of your HPC system is complete, for now. You’ll know the refresh has been a success if you see improvements in key metrics – throughput has increased, jobs running to completion and overall operations costs coming down. And most importantly, you should see greater satisfaction and fewer complaints from your end user.
Celebrate when you are done as this process is not easy – it often feels like replacing the tires on a moving car! But is that not why you enjoy the HPC world? It is not for the faint of heart. Nice work! Now lather, rinse and repeat. Happy clustering.