Download this case study as a PDF
The highlight of our relationship with Nor-Tech has been their ability to accommodate our increasing GPU needs in an efficient and sustainable way.
Gonzalo Merino, University of Wisconsin, Madison
Their Challenge
The IceCube Neutrino Observatory, which examines the cosmos from the vantage point of the South Pole, is the world’s largest neutrino detector. It tracks these particles from the most violent astrophysical sources: events like exploding stars, gamma-ray bursts, and cataclysmic phenomena involving black holes and neutron stars. Its goal is to answer some of the most complex physics puzzles to date. About 300 physicists from 45 institutions in 12 countries make up the IceCube Collaboration. The University of Wisconsin–Madison is the lead institution, responsible for the maintenance and operation of the detector.
The IceCube observatory was finished in 2010; now researchers understand the detector and how it functions well. They are currently in the process of analyzing and interpreting the data in order to help validate or invalidate theoretical models.
It’s not surprising that this project requires an enormous amount of data collection, processing, and storage capacity. While IceCube researchers have been confident in the project’s ability to solve physics enigmas, they relied on Nor-Tech to give them the processing power and storage capacity to do that.
Gonzalo Merino, Computing Facilities Manager, for the University of Wisconsin, explained, “Our role as far as computing services for IceCube is mostly for data processing and data simulation– processing the real data that streams in continuously from our sensors. We extract scientific results from that data and compare it with models and simulated data. In order to do that, we need a lot of computing power. Our main challenge involving Nor-Tech was that we needed help building custom servers that would be optimized for our simulation needs.”
He continued, “There are a lot of factors that play a role when simulating the interaction of particles with a complex detector and we had to build a model of our device in a computer that accounts for all of this. The simulation is essentially a computing program that contains the theory models and mimics nature. It allows us to validate and calibrate the information we are getting back from the sensors.
Nor-Tech Account Executive Tom Morton has been working with Gonzalo and other IceCube personnel for more than 5 years. “We have quite a bit of experience with research facilities,” he said.” The University of Minnesota was one of our first clients and then it just grew from there. In the case of IceCube, they needed high performance compute nodes, using off-the-shelf GPU cards with very high density. Ironically this is not something you can get off-the-shelf. They were looking to us for the expertise and resources to design what really amounts to supercomputing capabilities without the traditionally high supercomputer expense.”
Our Solution
Knowing how much power they needed, Nor-Tech engineers brainstormed a solution that was cost-effective without compromising capacity. “We developed and customized the product to be able to house 8 GPUs per compute node,” Tom explained. “They were looking for inexpensive GPUs because their code doesn’t benefit from and doesn’t require the more expensive GPUs. They needed to find someone like Nor-Tech that could integrate a rack-mount solution using cost-effective GPUs. It was a breakthrough for them to be able to get GPU nodes for a price point that is unheard of.”
Gonzalo said, “To me the highlight of our relationship with Nor-Tech has been their ability to accommodate our increasing GPU needs in an efficient and sustainable way. There were other options that we could have pursued, such as enterprise-class servers, but we would have been paying a premium for features that we didn’t need. With Nor-Tech we were able to exploit consumer-grade GPUs, which were fine for our workloads, but are not easily accessible in the market. These machines were not off-the-shelf, so there was a learning curve. There was an initial phase of R&D and they worked with us to optimize the configuration. That is the added value with Nor-Tech.”
Their Success
Over the years, IceCube has purchased a significant amount of GPU nodes, storage servers, and regular compute nodes. For every order, Nor-Tech conducted code-testing and sent a technician to help UW personnel integrate the equipment into their datacenter.
“Their capability of customizing the systems to our needs was important,” Gonzalo said. “I know that Nor-Tech does a lot of work behind the scenes to ensure that our system is always optimized. It provides us with an enormous amount of computing capacity for the budget that we have and I have to say that we are very satisfied. We have not had any performance issues and really no issues at all. Nor-Tech is a key partner for us.”
Gonzalo added, “We have recommended Nor-Tech and will continue to do so. In our community—there are 40 institutions working on this IceCube project. Every time I hear about other experimental physics installations with similar challenges, I do mention that we’ve had a very positive experience with Nor-Tech and their clusters.”
Tom summarized, “They relied on us for price-performance and that’s what we delivered. Any organization with applications that don’t require ECC memory and applications that don’t require double precision are excellent candidates for this solution.”
About Nor-Tech
Winner of Microsoft’s prestigious Most Valuable System Builder Partner award, Nor-Tech (Northern Computer Technologies) is an industry-leading technology builder and reseller best known for providing turnkey, people-friendly high performance computing (HPC) solutions and Ansys HPC integration. In addition to HPCs, their custom technology includes workstations, desktops, and servers for a range of sectors including computer-aided engineering (CAE) and computer-aided design (CAD). Nor-Tech’s engineers average 20+ years of experience. They have been in business since 1998 and are headquartered in Burnsville, Minn. just outside of Minneapolis. Clients include some of the largest organizations in the world. To contact Nor-Tech call 952-808-1000/toll free: 877-808-1010 or visit nor-tech.com.