UWM Case Study
Companies like Nor-Tech can beat much bigger competitors with a significantly better price and comparable or better quality.
-Tom Downes, Senior Scientist, University of Wisconsin Milwaukee Center for Gravitation, Cosmology, and Astrophysics
Project Background
At a national media conference on Feb. 11, 2016, National Science Foundation (NSF) researchers announced the first direct observation of a gravitational wave. This was a phenomenal achievement that resulted in a 2017 Nobel Physics Prize for LIGO project leaders. 1 Subsequent gravitational wave detections have confirmed those results. As the LIGO project continues to be refined, the rate of gravitational wave detections is expected to escalate to as much as one every two weeks.
Recently, the two LIGO laboratories that were the sites of the first gravitational wave detection–
Livingston, La. and the Hanford, Wash.–were designated Historic Physics Sites by the American Physical Society. Plaques were placed at both locations memorializing the extraordinary efforts
that ultimately led to what is widely heralded as the most significant scientific discovery in the last 100 years.
A team of UWM physicists, led by Patrick Brady, continues to play a key role in the LIGO project. At UWM they design, build and maintain computational tools, such as Nor-Tech’s HPC technology, that handle LIGO’s massive amounts of data.
In addition, NSF awarded the North American Nanohertz Observatory for Gravitational Waves (NANOGrav) $14.5 million to create and operate a Physics Frontiers Center (PFC) at UWM. The PFC, which is also using Nor-Tech HPC technology, ensures that researchers have the resources necessary to explore one of the most exciting frontiers in physics and astronomy.
Their Challenge
Nor-Tech has been providing HPC technology to UWM since 2005, which is close to the time that Nor-Tech was awarded a contract by the State of Wisconsin. Tom Downes, a Senior Scientist at the University of Wisconsin Milwaukee Center for Gravitation, Cosmology, and Astrophysics contacted Nor-Tech Senior Account Manager Tom Morton in 2017 for an update to their existing high performance Nor-Tech HPC technology. In addition to beating large competitors on price, UWM’s previous purchases provided assurance that the project would complete successfully.
“We have worked with Nor-Tech in the past and have had a good relationship with them,” Downes said. “And their business status with the State of Wisconsin continues to be important.”
The processor update was completed in January 2018. “They were using last generation Intel processors,” Morton explained. “They needed additional processing power using the latest Intel Skylake CPUs.”
Why Nor-Tech Clusters Use Only Seagate Drives
Nor-Tech used Seagate drives exclusively for the ORCA build. In fact, the firm uses Seagate drives for every build unless a customer insists on another manufacturer. After years of testing components to near failure, Seagate drives came out on top. For every high-performance computing cluster, Nor-Tech builds a test cluster for evaluation. “We’ll send the customer several nodes and they will test for two, three months, just heat those machines up and work on them to make sure they’re going to be bulletproof,” explains Bollig. “Once they determine we’ve created a bulletproof unit, they’ll go to bid.”
This process has given Nor-Tech a strong reputation for ruggedness and reliability. “High end clusters run at 98% utilization, and they’ll run like that for years,” says Bollig. “You need rock solid parts, and we’ve found that over time and the hundreds of clusters we’ve built, we always have our best luck with Seagate. When it comes to our clusters, my engineers will use only Seagate drives.”
Seagate support has also helped maintain that reliability. Daninger recounts an experience with the Seagate support staff. Nor-Tech built a series of storage servers using Seagate drives for the University of Wisconsin to help support the IceCube Neutrino Observatory. Due to the unique demands of the project, some of the drives were experiencing intermittent issues. Daninger called Seagate support.
“The field engineer from Seagate, Joel Schulze, took the ball and ran with it,” says Daninger. “It turned out that the firmware had to be updated on the drives. The Seagate support team wrote a script and emailed a link to my technician out in the field. He put the script on a thumb drive and that automated the installation, solving the problem all in one day for the customer,” Daninger marvels.
Our Solution
UWM’s Sandy Bridge processors were installed in 2012. They replaced Intel Westmere processors installed in 2010. “Every 3-5 years we go to NSF to ask for more funding to update our high performance technology with more contemporary processors,” Downes said. “This time we wanted to install the latest Intel Skylake processors. It was routine in that we want the technology to stay current for LIGO. If it doesn’t, it will become slow relative to the technology that others in the LIGO consortium are using. We also needed to be a failover site in case one of the LIGO data centers goes down—to take over if we need to.”
Downes relied on Nor-Tech’s experienced engineers, including Vice President of Engineering Dom Daninger, throughout the project and after delivery to provide expert advice. “Dom has always been very good at technical details,” Downes said. “If I need to know a very fine detail I know he will have the answer. I have always been satisfied with how responsive they are and the turnaround time for service and support is terrific.”
Although the project was fairly straightforward, Nor-Tech had to work around the usual budget constraints of educational institutions. “They came to us with a pretty tight budget,” Morton said. “Dom and I worked with Tom Downes to help determine which CPUs, memory and storage options would best fit their budget. We also worked with Intel for special pricing. In addition, UWM needed to work within space limitations, so we designed and developed a uniquely configured high density computer that our competitors could not offer. To accomplish this, there were several meetings at UWM that involved high level strategizing.”
Their Success
UWM is in the process of moving all of the LIGO-related technology to a new datacenter. In its new location, the technology will continue to analyze the data that UWM is getting from the instruments at the two primary LIGO sites in Washington and Louisiana.
“They have staff that regularly travels to Wisconsin,” Downes said. “If there is an issue, they will send someone over. For this particular purchase someone was onsite to help us rack the equipment and get rid of all the boxes. The boxes may seem like a small thing, but it can actually take an entire day; Nor-Tech will always go the extra mile.”
The new Skylake-equipped computer is proving to be almost 10 times faster. “People are just starting to use it and the results have been phenomenal,” Downes said.”
Morton continues to follow up with UWM and Downes to ensure an optimal experience. “Everything is running well,” he said. “Tom Downes and the entire UWM team are always a pleasure to work with. We look forward to many more exciting discoveries at UWM and are thrilled to play just a small part.”
Downes concluded, “ I have recommended Nor-Tech to others. On campus sometimes purchasing processes go through various levels. Most people are familiar with Dell and Apple products. One of the reasons I wanted to go through the competitive bid process on this was to show that companies like Nor-Tech can beat much bigger competitors with a better price. And that was the case here; Nor-Tech offered a significantly better price than their major competitors and there was no compromise in quality—if anything, the quality was better.”
