ASHRAE Journal:
Today's episode is powered by NIMBUS and their VIRGA hybrid adiabatic cooling systems. Built to last with minimal maintenance, VIRGA delivers lasting ROI, cutting water use by up to 95% and energy costs by up to 50%. Smarter cooling, stronger returns. Learn more at nimbus.cool.
Justin Seter:
We're going to go ahead and kick off ASHRAE Journal podcast, Liquid Cooling and Data Centers, Part Three. Please hold your questions till the end. If there are any, we'll try to catch you then or afterwards. So we'll start with some introductions here and then talk about the themes of 2026. For those of you that don't know, this is our third version of the ASHRAE Journal, Liquid Cooling for Data Centers podcast. So if you go back about six months and another six months before that, you can hear part one and two with my esteemed panel, and we'll let them introduce themselves here. We're going to go ahead and get started with intros. So my name's Justin Seter. I work with DLB Associates, Strategic Initiatives officer, as well as our affiliate companies. I've been in the industry for almost 25 years and almost 100% of that time have been in data centers, starting with commissioning, project program, people management, and now think a lot about what comes next for the industry, which is spoiler alert, liquid cooling and high density AI. So I'll go ahead and pass to Dave here.
David Quirk:
Welcome everybody. Dave Quirk, DLB Associates. I'm the CEO. I'm also the co-founder, president and CEO of several of our affiliate companies, including Fluid to Chip and others. So welcome to today's podcast. I have been in the industry over 25 years, heavy participant in ASHRAE, OCP, NFPA and all things EIEIO in the industry to help write the book on data centers. Welcome.
Dustin Demetriou:
How do I top that?
David Quirk:
It's all you, Dustin.
Dustin Demetriou:
All right. I'm Dustin Demetriou. I'm the chair of the ASHRAE TC 9.9 IT subcommittee. Get a little closer. IBM, I lead thermal engineering for our systems. Been in the industry for 15 years, much of that time doing liquid cooling systems within IT equipment, a lot of background in thermal design of IT equipment, data centers, et cetera. So thanks for having me again.
David Quirk:
I got to add to that. So he's leaving out the best part. You worked for the godfather of liquid cooling and data centers, Roger Smith. Come on.
Dustin Demetriou:
That is true. Everything I learned, I learned from Roger, so invented it.
David Quirk:
He was underselling himself there. I had to help out.
Dustin Demetriou:
Nice. Yeah, appreciate that.
Justin Seter:
So it's February 2026 as of recording date. So let's talk about where we are today, maybe a little bit about where we were last year, the last time you heard from us. My theme for the 2026 is, boy, have we moved fast. So last year there was a whole lot of talk about what was about to happen. And guess what? A whole heck of a lot of that happened last year. And so now as we get into the implementation phase and eventually get this technology up to maturity, we've uncovered a lot of stuff that needs a whole lot of work in the meantime. So that's kind of where I'm starting with my theme for the year, but I'll go ahead and let's let Dustin go first on his.
Dustin Demetriou:
All right. So theme I have is that I think the industry is starting to recognize that we're not in a three to four year life cycle of IT equipment anymore. We're in a one-year life cycle of IT equipment. And so what does that mean? It means there's no constant. Everything is going to be changing every year as we go on into the future. And so as I think about this, clearly what we're doing with AI and what we're doing with the hyperscalers and scaling liquid cooling is important. I think as important as we go this year and get to the end of the decade is how do we also enable the rest of the industry that is also going to have to move to liquid cooling? And so I think that's my theme. Every year we're going to have something different. We're going to have to think about how do we standardize and make things as repeatable as possible, but its speed is the theme.
David Quirk:
All right. I'm going to add to that. I'm going to pick off the second leg of the stool. So we got speed, cost and quality, right? Pick any two. And I'm going to pick the quality leg. I think those need to be the two that we focus on right now. And what we're seeing in the market is that there's a lack of focus on the quality. And why does the quality matter now? Well, we got process cooling loop in the mix. We have one of the most critical process cooling loops in any application in the world, because it's now supporting these tens of millions of dollars worth of servers downstream. So it's super important. We've also increased the degrees of freedom. We've reduced the safety factors in the systems and we've increased the transients and the transient response requirements. And so now you add all those together and they all lead to one road, which is we need to up the ante on quality big time. That's my theme.
Justin Seter:
Sounds easy enough.
David Quirk:
Go fast, do it better, but still keep it cheap.
Justin Seter:
No problem. No problem. So maybe we'll stick on that for a little bit and talk about what are we seeing on sort of scheduled pressures that—Obviously everybody sees a deal that gets announced every week in the news at this point of some gigawatt plus $100 billion thing somewhere. But the timelines on those, let's talk about that a little bit and then how that impacts the two legs of the stools that you mentioned there, Dave. Can you still go fast and have good quality? And if not, why not?
David Quirk:
Yeah, it's an interesting dynamic in the market. Megawatts are being bought in 24 month blocks. So it's kind of like how grocery stores buy beef. They do not buy it for next year. They only buy for next week. And so the data center industry is no longer buying their data center capacity out three, four, five years like they used to. They're only looking out 24 months. And it all derives from what Dustin said. The software and the hardware are changing so fast now. Nobody wants to commit their capital beyond that timeframe because it could all be changed again. So this is the new paradigm. This is the new world order in data centers is that we have to meet those timelines. When LOIs are cut, you have to get to that capacity because there's SLAs tied to it, there's penalties, and we're missing out on turning ones and zeros and making those into dollar bills within those sites. And that's all that they care about at the end of the day.
So what that means as a design engineer, commissioning agent, contractor, equipment vendor in the space now, we have to compress everything we do into that schedule, which means we can't do it in the same serial fashion that we used to do. We have to take things out of order. We have to get comfortable with taking them out of order and we have to do parallel processing like what we do in some of the CPUs and GPUs out there, right? So parallel processing. So we have to change forms of agreement. We have to procure things way before we even have a design and we have to get comfortable with wrapping a design around that. We have to change everything. That's how I'll leave it in order to go that fast.
Justin Seter:
Sounds pretty straightforward. Dustin, why can't you get these IT guys to stop changing everything so fast?
Dustin Demetriou:
It's funny when you think about AI, right? Just what we're doing with AI, right? What we're doing today is completely different than what we did a year ago, right? So a year ago, what were we talking about? We were talking about these large language models. It was like, "Let's chat with this thing." So what are we doing today? Well, today though, it's the world of agentic AI. So what does that mean? It means we need a lot of these large language models that all talk to each other, communicate with each other, and then also we're going to be able to automate lots of things, right?
So it's not just that the hardware is changing, it's that the application of AI space are evolving even quicker than one year. And in some cases, we're building applications that are making the hardware that we just put out six months ago obsolete. Because you need more GPUs, you need more memory, right? As we have all of these models that have to talk and communicate with each other, it just drives the scale of infrastructure that we need. And so I kind of look at this from the application space, so why can't we keep things constant in the IT? It's because we trying to meet the demand of the applications that people are trying to do with these things.
David Quirk:
It leads to my favorite question. Who is the occupant of a data center? Anybody know? Anybody? Bueller? Software. Software is the occupant. And if you think of it from that lens, as Dustin was just saying, how quickly does software change? How quickly do the applications change? And that's how quickly we need to start thinking about changing the infrastructure in these sites at the speed of light.
Justin Seter:
Chilled water plants operate at the speed of light, right? So with all of the-
David Quirk:
Buffering. Buffering.
Justin Seter:
With all of that rapid change then, how does that play into the project life cycle? Because you'll call your chiller manufacturer today and tell him you need 10 or 100 chillers or whatever, and he quotes you, I don't know, 12, 14, 18 months, something like that. In the project life cycle then, knowing that the point of the project is to serve the software and if you have to have different things to serve the software based on the IT generation, how in the world do you ever strike something in the sand 24 months in advance and say, "We're going to build this optimized design that's perfect for this application." Is that even possible?
David Quirk:
Is that a question? Of course it's not. Why? Because we're still trying to do it all the way we did before. We are still trying to do design, bid, build. The bulk of projects in the industry still do design, bid, build. That process for a typical data center does not get done in 24 months when you're trying to stand up 400 or 500 megawatts in a building and multiple buildings on a campus. You're not going from the point of design, say go and get that complete, get everything ordered and get all that constructed and turned over and commissioned. And let's add the fact that it takes way longer to commission liquid cooling. We'll come back to that later. In 24 months, it's not happening.
Justin Seter:
Okay. That's a problem to solve, but stuff's getting built today. It's happening. So what is happening on the ground today then to respond to that for things that started getting designed a couple of years ago? As I mentioned, a whole bunch of projects came online last year. I don't know how many people listening were involved in some of those, and there's a whole heck of a lot more that are in flight right this moment. So what are we seeing out there? Is this schedule pressure driving everybody's good behavior?
David Quirk:
No, it's bringing out the worst in us. So we are skipping steps. That's what we're doing. Back to my quality theme. And I think the industry's getting a real wake-up call. We're seeing it on a lot of projects. We're getting to the final yard line and then realizing, whoops, we treated the TCS like the FWS loop, or the technical cooling system like the old chill water pipe system. It does not work that way. As I quoted before in one of the industries, you got to be able to drink milk out of that pipe on the TCS side. It's got to be that clean. And so we have to change the mindset, we have to change the players, we have to change the quality control, we have to change the processes, we have to change the schedule involved with making all that come to be. And just to fill and flush those systems, just to get the air out of them takes days, if not weeks, just to get the air out. And if there's any air left, guess what? You're going to roast some servers, right, Dustin?
Dustin Demetriou:
They are going to overheat quickly. The other thing I'll add to that is I think going back 18 months ago, there was a thought that we could transition to this need to get to full scale liquid cooling with things like liquid to air CDUs. So we have sites that don't have capabilities of liquid today, water, et cetera. So we're going to have a liquid to air CDU and just reject the heat. Well, we're past actually being able to even use that technology to transition us, right? The demands of the airflow that are required in those types of systems are well beyond what you could do. So it's not even a choice really to use technologies that we thought were going to be stop gaps for the next five years to get us to full scale liquid cooling. These technologies are obsolete already. So again, 18 months. 18 months ago, we thought we had a great solution. Today we don't have a good solution except for go build lots of liquid cooling, right? Full scale liquid cooling.
David Quirk:
And we're already seeing the limits of even the air cooling solutions we built for these sites running up against the limit. So average cloud site today, 10 kilowatt of rack. We're using fan walls and craw galleries at the end. Well, they have a limit of the throw of the amount of CFM they can get down that aisle. And when we're a hundred kilowatt of rack on liquid cooled, which is like a small density these days, I hate saying that already. We already blew way past that. If that's already at 20 kilowatt of rack for air cooling.
And so now we're having problems with how do we get all that air to the racks that are at the end of the lineup or in the middle of the lineup, depending on whether we're single or double ended. So yeah, to your point, by the time we've designed all these new technologies, we've already outpaced it with the next evolution of the chip because as they just announced at the CES conference, we're already pushing four and 500 kilowatts of rack. And by the way, there's already one megawatt racks out there for those that hadn't gotten the memo.
Justin Seter:
So not a trivial challenge to solve then in those cases where not ideal design is being adapted for hardware that's changing on the fly. So what framework do we have then for people to reference things like design standards and acceptance criteria and these kind of things? I mean, if we go back 20 years to when ASHRAE TC 9.9 was founded and got all the IT manufacturers together and said, "Hey, let's all agree on these environmental boundaries for air cooling and things like that." And we got there and that was a game changer for the industry and that kind of thing. Do we have anything like that for liquid cooling or how far off is it?
Dustin Demetriou:
Yeah, so I mean, we have some of that, right? So we do have thermal guidelines. So last year we introduced the TCS thermal guidelines. So temperature rating classes for that fluid temperature going into the equipment. So we have some things like that, that's constantly changing too. We just heard announcements where a year ago we were talking, maybe we need to go to 25, we heard an announcement, maybe we need to go to 45. We, TC 9.9, put out some guidance around trying to get the industry to use 30 degrees Celsius as kind of even playing field. So we have some things like that that are starting to be adopted. I think as a thermal engineer, that's great. I love that we have that kind of stuff. I think the big gap we have is on fluid quality, materials compatibility. This is the biggest gap in the industry today.
We have a multitude of fluids, a multitude of materials, right? We have designs where, yeah, we spec it with this material and we actually go and do the job and they put another material in it and it makes it completely obsolete with the fluids that are going in there. And so to me, this is the biggest gap we have. So we have some material that's out there from TC 9.9. We're doing some partnerships with folks like the open compute project to try to kind of see how do we—kind of commonalities across these different companies. But the problem is it’s just a multitude of fluids. A multitude of materials. Everybody's thing is better than everybody else's thing. And until we kind of get that kind of even playing field where we're all designing and doing the same things, it's really hard to standardize any of this stuff, right? So to me, fluids and materials is the biggest gap we have in the industry today.
David Quirk:
So we have this new process application called the technical cooling system and data centers, liquid to chip. And we have no standard of care, we have no processes for it, and we don't have agreement on the materials, the temperatures, the fluids. What could go wrong? Yeah. A lot could go wrong as we've seen already in the market. So first things first, what ASHRAE TC 9.9 is trying to get out to the market is to raise awareness on all these items. So there's online encyclopedia that has a bunch of publications. We're putting out tech bulletins as fast as we can because that's the quickest way to get the news out to the street. So if you're not already following the tech bulletins that have been published and part of the online encyclopedia, please do so. That's our plug there for the ASHRAE online encyclopedia. We are also conducting a lot of research within the committee.
Dustin Demetriou:
Yeah, for sure.
David Quirk:
So I originally authored one of them. It's Work Statement 1972. It's a research project to help identify what's the acceptable rate of rise for the liquid going to the chip. And for those that have been around long enough, you know we had an acceptable rate of rise, right? We had a boundary condition for the air temperatures going into the server, but we also said how fast you could change that. Go too fast one way or the other, you can create problems, thermal stresses on the boards and all the connections and ultimately failure of those servers. The same is true with liquid, except it happens much faster.
The time constants are very different for liquid versus air. And so this is a really big deal. And right now we don't have any guidance to the market, but we do have a project that was just recently awarded and so we're off to the races to get some answers. That project's going to actually review things like going up to 85 degrees C at the junction temperature and/or when do we cause throttling on the server and we'll classify those as failure conditions and then use that data to back into an acceptable rate of change that we can change that temperature. That's a really big deal for anybody that designs, commissions, operates data centers out there in the liquid to cool space. What else we got?
Dustin Demetriou:
Yeah. So a couple other things that are in the works, other things that we need to tackle. Again, doing this collaboratively with Open Compute and TC 9.9, right? So two big things. One, fluid mixing. So you have two different propylene glycol-based fluids. Can you mix them together? Who knows? Nobody has any idea. So what happens when you have a system out there, you've brought it up and you use fluid one, and then you can't get it. A supply chain's broken and you have to get fluid two. What do you do? You drain the entire system and fill it back up? Who knows, right? Or what happens when you test a system with one fluid and you don't drain it all out and you bring it in, you bring it up and you put a different fluid in there, right? This is a big issue. Can you mix different propylene glycol-based fluids together? They have different corrosion inhibitors, different things in them. This is not obvious. There's no answer out there today. Sounds like you have something you want to say, Dave.
Justin Seter:
Yeah. I mean, historically everybody's just said no. You can't mix them, right? It's similar to like if you've got a European car and you pour in the orange coolant, or if you got a Ford and you pour in the green coolant that you buy at AutoZone, those two fluids are very chemically different. And so if you get some in between or some mix, who knows? Everybody just says no. But to your point, we already know that the PG25 supply chain in the nation right now, today, is stressed. They can't produce enough of it. Price is going up, lead times are going up. PG25, by the way, is the agreed upon fluid of the future, not in all sites. There's still some sites that do deionized base water and things like that, but we need to understand that because there's systems that get turned up, as Dustin said, and then IT, some of the IT rack manufacturers prefill their racks with PG25. Historically, we've said it better be the same one, but is that going to be the case in every example we have here.
David Quirk:
It's already not.
Dustin Demetriou:
Yeah. And it's more than that, right? It's like when we design the IT equipment, when you're thinking about safety testing that equipment. Today, we have to safety test equipment with the fluids that are going in there. So how do you even safety test and service equipment if there's a hundred different fluids out there? This is a problem. We end up with equipment that sits in compliance testing for months and months and months and months because we have to do an infinite number of materials, compatibility tests, fluid compatibility tests, et cetera. And so even from the IT design side, we do a lot of work to qualify all of the materials that are in that system and we qualify it with a certain fluid. Well, when you go to these large hyperscale applications, sometimes they pick their favorite fluid. And can you really just plug in that equipment into a system that has a different formulation of propylene glycol fluid? To me, if you haven't tested it, you have no idea and we can't test everything.
David Quirk:
Wetted material compatibility, big problem. So we talk fluids. What about the metals? What about the metals and the fluids playing together in this same swimming pool? What about when we mix our metals? So we got copper, we got aluminum, we got carbon steel, we got stainless steel, we have bronze, we have zinc in it, la, dah, dah, dah, dah. Oh, do those all work well in the same swimming pool? Maybe, maybe not. Depends what kind of fluid we have in there, right? What temperatures are we operating at? So on and so forth.
Dustin Demetriou:
Yeah. So on that line, I mean, the other piece of research that TC 9.9 is kind of working on is velocities. So we know velocities can cause erosion, corrosion. What velocity does that happen at? I don't know. Nobody knows. There's some guidance out there in the industry for things like staying below certain velocities from an energy efficiency perspective. There's some research out there that has looked at this, particularly in more corrosive environments where you have salts and things, but what about in these TCS systems? When you have these cold plates that have literally 50 micron channels in them, any little bit of pitted corrosion is catastrophe. If you have a big pipe and have a thick wall, like a couple of mills a year of corrosion is no big deal. In this cold plate application, it could be the end of the world.
Well, what are those velocities that we need to stay below? Are they dependent on the fluid? Are they dependent on the material? And again, this is another question that the industry doesn't have a good answer to. And again, we're working with OCP to try to establish that boundary condition, because the one thing we do know is that the flow rates that we're at in these systems today are not sufficient for what we need to do into the future. So flow rates are going to go up. I don't think we really have the opportunity, particularly in the server to necessarily grow the pipe size to keep the velocities low, right? So we're going to be putting a lot more fluid through the same size pipe, means a lot higher velocities that we're going to have to deal with.
David Quirk:
Speaking of velocities, we just got some answers here recently out of a research project on what are the acceptable velocities in order to flush the system before we do fill it up with the glycol, right? And those velocities are not trivial. We're talking anywhere from four, seven, 10 feet per second. We're getting to some pretty serious velocities. Are the CDUs even going to be able to do that? I don't think so. Houston, we got a problem. So we have issues on the velocities on the flushing side. We have issues on the velocity on the operation side from erosion perspective, but guess what? We don't even control velocity in these systems at all. It's not a consideration, right? We're controlling delta-P. We're trying to control temperatures going through servers. So is anybody even on first? Are we even paying attention what the velocities are for that? Who knows?
Justin Seter:
No problem. We got this. So a whole bunch of these projects that are in flight now are some combination of all of that, right? Different fluids, different velocities. The cheat sheet for what Dustin said, I think is really important. So I'm going to repeat it. As the densities continue to go up, so as everyone hears the next big density announcement, you kind of got two options for more density in the same footprint. One is you got to get more fluid, so the flow's got to go up. And in these cases, the cold plate size is relatively fixed by the form factor of the U of the servers themselves. Or the temperature has to go down.
I know everybody's super excited about hearing that the temperature get to go up and there could be some scenarios where that may be the case. There's also going to be scenarios where the temperature has to go down just because the density in the same footprint and will be at the flow limit or whatever it may be. So let's talk a little bit about that. Let's hit on the 45 C thing here real quick. And what do future systems look like? So again, remember, we have to sign the deal 24 months before we energize or less. So do we always know when we sign that deal what the IT is going to be in that room?
David Quirk:
All right. I'll take the crack first. I think about 18 months ago, I said we were going to go to a four pipe system. Last year I said we were going to go to a six pipe system and I'm calling it right now. Eight pipe system is coming.
Justin Seter:
Great for the piping guys, by the way.
Dustin Demetriou:
Great for the pipe industry. Yeah.
David Quirk:
So why the multi-pipe systems? Well, because we have multi-cooling requirements within this space. So we're going to see a departure right now between the air cooling and the liquid cooling. We're going to have a lower temperature class probably typically, but not always, I'm going to caveat, not always, on the air side, but a higher temperature class for the liquid cooling. And what we could see, and likely, and that's what I'm claiming here, we're going to see a mix of sites that have 25PG or DI water and refrigerant loops in the mix going to the data floor and a different temperature class altogether, hence the eight pipe system.
So you're going to start seeing sites with multi-pipe arrangements, all serving different servers on the floor. And why would that happen? Well, I don't know. The hardware's changing every six to 12 months now. That's how. So by the time you built the building, whatever you build it for is already changed and it's going to change again in another 12, another 24 and so on and so forth. So I do see eight pipe system is going to become the new reality. And if you're going to ask me what's next year look like, what's next year look like? 10 pipe systems coming, I'm telling you.
Justin Seter:
More pipes is more better. So one of the reasons I wanted to hit on this, and Dustin, I'll let you jump on that one too. I wanted everybody to think about this concept of like where we lived for a long time in the air cooled server world. It doesn't look like we're going to get there anytime soon in the liquid cooled world. There is too much variation. And in fact, the opposite has occurred to where not only do we not have convergence around a single temperature environmental requirements, we have accelerating divergence in different directions, hence the multipipe system. Go ahead, Dustin.
Dustin Demetriou:
No, yeah, no, exactly. I mean, so this notion of 45 C temperatures, warmer temperatures is not new, right? I mean, actually TC 9.9 published the W classes, I think it was in 2010 or something like that, where we introduced this concept of higher temperatures at the time, really built around waste heat reuse and applications like that. At that time, we were literally cooling 90 watt processors. Today we're cooling 2000 watt processors. So is it a reality for today? Yes, in some cases it can be. There are consequences of that though. So if you have the 45 C water loop or fluid loop and you have a 20 degree room, what's going to happen? It means you're going to be still rejecting a bunch of heat to the room.
So you're still going to need air cooling. I look at devices like storage devices, I look at networking equipment. All this other stuff that for the most part today is not liquid cooled at all. There's examples, but it's not—is this thing on? But that equipment is not really liquid cool today. So you have to be able to support air, liquid, different servers, densities are going to require different temperatures. There is some truth to some of the things you hear out there. Lower temperatures means you can probably run those chips a little bit more power. You're going to get more performance. So these are kind of the trade-offs that manufacturers have to go through. And there's not one answer.
Every application and every server design is going to be specifically designed to get the most performance for the specific application while driving efficiency, while reducing cost, et cetera. And so is 45 C real? Would I stake my design on only supporting a 45 C system, not put any mechanical refrigeration in and think you're going to be good for the next 15 years? I wouldn't stake my career on that. So this is kind of the world we live in just because of the rate and pace that things are changing. And back to the comment I made earlier. Two years ago, TC 9.9 tried to kind of put a stake in the ground around, could we establish 30 degrees C as sort of a standard condition that people design around to? And I guess we've kind of failed because it hasn't worked.
David Quirk:
Standardization, what is that? We can't even agree on like 25% polypropylene glycol yet.
Justin Seter:
So what we're hearing so far, we're about a half hour in here and we've got another 20 minutes or so to chat. And so I think where I want to head with this next is all of that stuff sounds really complex that we just talked about. Not here to scare you, but it is complex. Call to action real quick for the industry. Can we just go ahead and stop trivializing that it's complex? And if I hear one more person say, "Oh, it's just a pipe or it's just a chill water plant or it's just, it's just, it's just." No, it's hard. It's hard because the conditions of the projects are changing, all of them.
Infinite degrees of freedom, new players in the industry, no standardization or regulation on the designs, quality requirements and acceptance criteria. Flush and fill alone is probably the single least understood part of how to do a TCS system, which we have some research potentially in the works for that as well. So this trivialization of we can do this thing that's 10X or 100X more complicated, faster, cheaper, and still keep good quality, no problem, right? So that's where we are. But what do you guys see is like, how do you control some of these degrees of freedom to try to like give yourself a chance to succeed?
David Quirk:
ASHRAE, right? We get the group together and we get a common understanding and we establish some ranges, some acceptable practices, just like what we're doing with OCP right now as well. And we put the publications out, we get the industry to react and then we adjust and we update. But I think we're in this moment in the market where we have a great divergence occurring right now. It's the wild, wild west where there's lots of opportunity for innovation and that's wonderful. We're seeing tons of that on all sides, but I'm going to go back to one fundamental issue that's really important that we need to keep harping on. The occupant is software. Software changes really fast. The hardware's changing really fast. We've gone through now, what, three major chip evolutions just in the past 18 months, and the chip manufacturers have been slowing that down on purpose because the market's not been able to absorb it quick enough.
But each of those changes was a massive step function. We went from 10 kilowatt of rack up to 50 to 60 kilowatt a rack to about 120 a kilowatt a rack to now five to 600 kilowatt a rack and select applications over a megawatt a rack out there in 18 months. So I think the answer is we get more involvement from the industry on these committees to help come together with the ideas and standardize. We have to create some commonality, some common platform processes, acceptable ranges, et cetera, et cetera. So we can then all do our own individual proprietary thing, but we also have that common ground. And I think companies like NVIDIA are doing great at like setting the groundwork for the market on that. There's a place for open source, there's a place for proprietary. You can have your cake and eat it too, but as an industry, we need to create more of an ecosystem and more collaboration to get to the right answers here faster so we don't screw this all up collectively.
Dustin Demetriou:
I agree.
Justin Seter:
So now's not the time for protecting your secret sauce. There's plenty of money to be made. There's plenty of innovation still to be had in the industry. We need people to work together to come solve this problem to actually get it done. Because if we all focus on our little piece of the pie, back ourselves into a corner and say, "This is all us." At the end of the day, the client's not getting what they're looking for because they want the project to succeed. There's plenty to go around. Everybody needs to come to the table. These are volunteer organizations like ASHRAE and Open Compute that everybody's looking to lead this industry here and we need help. We need help. If you're a process piping guy who knows everything about ASTM, A380 and 967, stainless steel passivation and chemical treatment and flush and fill and all of these kind of things, these are the most important topics that our industry has no regulation on.
Dave made the dairy truck joke here earlier. Yeah, nobody's drinking out of this pipe, it's not going to kill anybody if your TCS loop's dirty, but we're talking hundreds of millions of dollars of connected equipment. And a hundred million dollars of the connected equipment is one row. So like now go ahead and make it a hundred rows and then make it 10 buildings and then make it a hundred sites. The numbers are astronomical, it's staggering. And so we need that kind of expertise in our industry. We don't have to overregulate it like it's a nuclear site or something like that, but we need more than we have right now. Right now we need to find a way to shoot the middle between having enough regulation so everybody's playing ball, but then still being able to go fast with good quality.
Dustin Demetriou:
Yeah. And I think to that, this is a theme we talked about at the recent IT subcommittee meetings is back to speed. How do we get to speed? Well, I think as engineers, we like to put the Ph.D. thesis down on paper. I think today, really what we need is the bullet points. We just need people to have the right talking points. Today, most of the times we don't even know the right questions to ask. You get a question from your senior management because they hear something. We need the bullet points. We don't need somebody to work. We don't need people to write lots and lots and lots of material. It's the bullet points.
And that's a theme we're trying to do back to the tech briefs that we've introduced, this new format of content. Yeah. Having the long form content is great, but really being able to get the material out, speed to market, get quickly, how do we do that? I mean, Justin made a joke recently around, we were aiming for, could we publish four or five of these within a year or six months just to get material out there? And Justin made a joke that said, "We really need like 40 of these within the year." So again, we need volunteers that have expertise on topics that can come and help and at least help the industry start to ask the right questions.
David Quirk:
I'll throw out a few right now here live. I got a few suggestions. I think we need one on what is—
Dustin Demetriou:
Are you going to work on them?
David Quirk:
Yes, I am going to work on the plane ride home, of course. I think percent polypropylene glycol. What's acceptable? Can we do 23%, 20%, 15%? Can we do 30%? Does it matter? Well, I think we would all agree it matters, but there's requests out there to do across that entire spectrum I just said. So I think just giving the industry, "Hey, this is where you should be." And if you go outside that range, you're on your own.
Here's another one. How long should you anticipate the time to do fill and flush, passivation in the field if you have any field welded stainless, or just how about removal of the air from the system? Can we just give like just a simple call out for how long it takes to remove the air from a typical CDU loop? It's unbelievable how long it takes to do that. And it's important that you get every ounce of it out and nobody's planning for it right now. I could keep going on with the list, but you guys get the idea. I think we need to add just some more quick bullets on that.
Justin Seter:
I think it takes longer if you don't design in air vents. So maybe we'll start there. Let's make sure there are air vents and ways to vent air from the system.
David Quirk:
You got me there.
Justin Seter:
But again, back to the basics. It's not a trivial exercise to think about from the design through the operations and who's the lucky operator who gets to plug a two-inch water hose into a $7 million piece of IT rack. All in between there, there's hundreds of stakeholders on these projects, hundreds. What is the chance that everybody's on the same page? I've been doing this for a long time, and so I'm going to go ahead and just propose zero. It's a zero percent chance that everybody's on the same page. So through TC 9.9, there's been some work in OCP, go follow some of these OCP work streams for modular TCS and things like that. We are working on some of this actively. Again, we need help. We need experts. We need people from the process industries that are used to doing semiconductor, biopharma, and the geeks behind the curtain, The Wizard of Oz kind of guys that are designing all of these chemicals and things like that, that are ultimately going into these systems. We need those people at the table.
David Quirk:
I thought we were the wizard behind the curtain. Anyway, and for all the people that are walking by right now thinking this isn't my problem to solve, it is your problem. Literally, everybody just walking past right now, casually on your phones, this is your problem. So we are literally calling on everybody in the industry to get involved with ASHRAE and OCP and the other organizations to help start closing the gaps because we have too many degrees of freedom, two little safety factors in the systems now, and we have a divergence of technologies. So we're going to see sites with multiple temperature classes, multiple liquids, coming to a site near you. And I got asked at the panel in that prior session that we were in, how many sites are we building now that are liquid cooled? All of them. All of them. Yeah, so this really matters. We need to pay attention.
Justin Seter:
All right. So we've got about 10 minutes left here. Let's make sure we have some chance for you guys to get in here. What are your non-negotiables these days? There's some stuff that's just not optional, right? Somebody's going to listen to this podcast and take away, I don't know, top three things they need to be sure that they get into their next project. What are you guys thinking about?
David Quirk:
I'd start with quality again. We need to double and triple down on the quality topic in general. And what I mean by that more specifically, we're back to the three-legged stool of price, quality and speed. And the market has already said speed is one of the two. There's no stopping that. So quality has to fall in line next because we are introducing so many new variables into the mix. Just to give a simple example, on the TCS loops now, we're adding thermal storage in the cooling systems. So whether it's on the facility water side or the TCS side, there needs to be this thermal energy storage in the mix now, and we're having to put all the pumps on UPS and we're having to run active redundant pumps in the CDUs. Why? Because if that flow stops ever, ever, we're going to fry a bunch of servers.
So that's the kind of criticality that we're now dealing with. Data centers, cloud-based data centers, they go down all the time, whether you believe it or not. And when they go down, they're able to come back online. With a liquid cooled data center though, if you have that thermal event that I just said, it's not coming back online. It's going down, it's staying down. So that's why I'm saying quality really matters now and we really need to change the game and our way of thinking and operating. And if you have any junk left inside that TCS loop when you plug in servers, there's a server guy going to be calling you pretty soon because it's going to clog that cold plate in a heartbeat and that thing's not going to be able to throttle up and do its job. And the IT guy on the other end is going to know it in about five minutes and they're going to be buzzing you.
Dustin Demetriou:
Five minutes?
David Quirk:
Okay. Five seconds.
Dustin Demetriou:
There you go. Okay. So you said you took two things. So I was going to say quality and then you actually talked about resiliency and so that was going to be my thing. But I agree. I think resiliency to me, thermal energy storage. On the application side, yeah, we can do things to shut down applications, do it gracefully, make sure we can save data and things like that. So in an air cooled data center, that was great. We lost cooling unit. We had minutes of time to do some of those things. We have five seconds of time to do those things if we don't have resiliency built into the system. Some basic things, and there's been research published.
We think we probably have talked about this on other podcasts, but it's worth mentioning again. You look at even something like a CDU, which could have redundant pumps in it. How are those pumps controlled? Is one pump running and the other ones shut down? And if there's a failure, that pump goes down and the other one starts to ramp up. That little blip, those seconds it takes for that other pump to come online are too long. So are we running all these units with active-active pumping schemes. Do we have backup power on these units? So to me, it's resiliency. It's kind of built on quality, but we really need to think about never letting that flow stop. The second that flow stops, the second all those things are going down.
David Quirk:
And let's not forget about all the thermal mass. So we've got a 4,000 pound cabinet that is at a much hotter temperature than what's coming out the back end of it in the room or on that pipe and that thermal mass is going to go someplace. So the moment that flow stops, there's a problem. And that room's got to absorb all that. The equipment's got to absorb all that, and it's all operating at a much higher temperature than what's in the ambient. Anyway, just food for thought on the resiliency topic. Yes, if this is keeping you up at night, it should. That's the purpose of this podcast.
Justin Seter:
Thought you were going to go to sleep and have sweet dreams after listening to the ASHRAE Journal podcast, you are incorrect. So my non-negotiable here for today is just data. Data, data, data. So we've seen some projects incentivized more by schedule than quality that make decisions that are not data driven. They're like, "Ah, is it good enough?" Yeah, we could probably get by. Is it good enough? What is good enough? Given the criticality of what we've just talked about, good enough needs to be good. There's no such thing as good enough. And so how do you get there? Well, if there's five, six, seven steps to initial fill, passivation, cleaning, secondary fill, final fluid fill, whatever it might be, how much data are you gathering at each step of that?
And is there a test result you can point to that says, "Look, I'm within parameters of this X, Y, Z and so we can move on to the next step." Or do you take that sample, ship it off to the lab, get your results back two weeks later and say, "Oh man, well, we went ahead and moved on anyway." Where's the chemical treatment testing lab designed into every data center now? Is anybody? Oh, they're not there yet. Oh, okay. Okay. Got it. They will be soon. So all of this data that's required is our only chance of holding the line on quality. And we've got to have owners that are ultimately paying the bill who want this thing to succeed to buy in on this concept or else it's just the same old game.
David Quirk:
My final non-negotiable is everybody's got to pay attention. Pay attention. So this stuff is super, super important. That's the call. It's a non-negotiable now and all the detail matters, all the detail.
Justin Seter:
Great. Okay. Well, I think we could pretty much wrap up part three here. For those of you that are listening online, we are live in the room at the AHR Expo here with probably about a hundred people and I see like at least five people who look like they have some questions. So we're happy to take a question or two. You can be on the podcast. Anybody?
Tom Corrigan:
Hey guys, really nice discussion. Thanks for that. So my name's Tom Corrigan. I'm with the PFX group. So we're a PG25 supplier, so keep that in mind. There's going to be a right answer to this next question. So you talked about the obsolescence of hardware, of software, this 24 or 12 month turnover cycle for technology, but you're talking about PG25 like it's here to stay. I only heard one reference to refrigerant in the TCS loop. Do you think there's going to be any disruptors in terms of the fluids, trends towards two phase immersion? Are those going to have a significant play in AI infrastructure going forward?
Justin Seter:
Yes and yes. All combinations of everything are going to be needed for essentially an infinite combination of projects that could be executed. So there is a density limit where we start to touch the top of what single phase liquid cooling as we sort of think about it today, kind of runs out of room after we've increased the flow as much as we can and made it as cold as we can. What's next, right? You need a phase change. So Dustin, I might let you talk about that a little bit more.
Dustin Demetriou:
Oh, yeah. I'm not throwing anything out at this point. I mean, everything's on the table. Yeah. So we're using 25% PG today. Obviously there are benefits of that and drawbacks of it. We've used water based systems for a long time, so probably a little bit more thermal performance there. Is it enough to not use the PG TBD? Today things are working. Yeah. Two phase probably gets us some more thermal performance. When do we need that? Nobody knows. So today, not throwing anything out, I guess is the answer.
David Quirk:
I think we'll see a mix of all of them for a long time to come. We're still using air cooling and we're still going to be using air cooling for a long time to come. I think we're seeing a whole new class of chip designs getting ready to hit the market and a whole new class of cold plate designs. So right now we have this simple linear channel type of flow configuration, but with 3D printing, we can do all sorts of new geometries and those will change. All the calculus we talked up here about what's acceptable in terms of the micron size for particles, what's the acceptable velocities for erosion.
As soon as we throw a new type of coal plate into the mix, everything's got to be relooked at top to bottom. I think we will have combinations of single phase and two phase refrigerants. I think we'll see a little bit more immersion on the scene, but immersion's a different operations model, different yield in terms of how much compute you get at per square foot in the building. So every business model's got to look at that performa differently and make that decision on which direction they're going to go. Obviously we've seen higher uptick with immersion on the crypto side of the world, but yeah, I think it'll be a combination of everything in the mix for a while to come.
Dustin Demetriou:
Don't stop making PG25, I guess, yeah, that's the takeaway.
David Quirk:
Wink, wink, wink, hint, hint.
Question Two:
Question is on that two phase trend as it interlocks with ASHRAE or code standards that would put requirements on ventilation in the white space, do you think that industry is going to digest ASHRAE 15, for instance, as it applies to ventilation rates with refrigerants in the white space, as well as it's adopted in the equipment rooms today?
David Quirk:
Yeah. I believe there's already some conversations amongst some of the TC 9.9 members and the group with SSPC15. I think there's probably not enough there right now. I think there's a general awareness that it can be a problem, but there's not enough application of refrigerant based solutions at the scale we're talking for it to be on the radar formally. I believe it will be within a year, and those conversations will probably result in some addendum to 15. And I think it'll also be up to a lot of the equipment vendors and the design engineers to look at where do we put the refrigerant in the heat rejection path between the server and the ambient. It doesn't have to be all or nothing. It doesn't have to be in the whole thing. So there's ways of compartmentalizing, putting in additional heat exchange methods like CDUs in the mix that will allow us to compartmentalize and limit the volumes of those refrigerants to where we can work within the boundaries of the current framework of 15.
Dustin Demetriou:
Yeah. I also think that we actually don't even know what the right answer is from a refrigerant perspective today. So there are refrigerants out there that work in applications. If I look at a server application, what are we designing to? We're designing to concurrent maintainability. So can you actually design a high pressure refrigerant system with some of these new refrigerants in a concurrently maintainable application? So I think there's still a question on what is the right fluid or refrigerant for these types of applications.
Justin Seter:
All right guys, we're getting a high sign here. So thanks to all who joined here in person and out here in the virtual world. I'm sure you'll be hearing from us again soon. Thank y'all.
ASHRAE Journal:
The ASHRAE Journal podcast team is editor, Drew Champlin; managing editor, Linda Rathke; producer and assistant editor, Allison Hambrick; assistant editor, Mary Sims; associate editor, Tani Palefski; technical editor, Rebecca Norris; and creative designer, Teresa Carboni.
Copyright ASHRAE. The views expressed in this podcast are those of individuals only and not of ASHRAE, its sponsors or advertisers. Please refer to ASHRAE.org/podcast for the full disclaimer.