Reliability means more than simply keeping systems running. It also means keeping systems accurate. Here are five industry stories that gave us the engineering discipline to become a more reliable partner.
By Rami Khouri, SVP of Engineering, Hyve Solutions
It's Halloween night. You're on call for a 4,096-GPU training cluster.
Every node reports healthy. Every GPU link and every NIC port is up. No driver faults, no ECC alarms, no thermal throttling. The BMC event logs are clean.
But the loss curve just jumped, and nobody on the bridge call can tell you why. The last three checkpoints might be good. Or they might not.
If you've been on a call like this before, you'll remember that sinking feeling in your stomach. Where’s the ghost?
This isn't a campfire story. Engineers at Meta and Google have published research on processors that return wrong answers with no machine-check exception, no ECC event, and no crash. Google's team calls these phenomena "mercurial cores." The symptom is simply a wrong number.
What keeps hardware engineers up at night isn't the outage everyone sees. It's the error nobody sees.
So for Halloween, here are five stories, one for each phase of a platform's life: screening, pre-design, design, validation and ramp. Four of the stories come from the public record. One is a composite of every hardware team's deepest fear. In each case, I'll walk through what actually broke, what it cost the people who had the most at stake, and what practices can prevent future nightmares.
One thing up front. Validation matters a lot, and I have deep respect for our validation teams. But no amount of validation can save a bad design. Reliability starts long before the first board reaches the bench.
1. The files that vanished
Lifecycle phase: screening from factory to fleet
What happened
In 2021, Meta engineers published a case from their own fleet. Files were stored, compressed, and decompressed on request. One decompression step computed a file size using Scala's math.pow function. On one specific core of one specific CPU (core 59), Int(1.1^53) returned 0. The correct answer is 156. Change the exponent to 52, and the same core correctly answered 142.
A file size of zero meant decompressed files came out truncated, with no end-of-file markers. The results were dangling file nodes and missing rows in the database. Nothing in the hardware raised a flag. The application eventually reported data loss, and Meta notes that this class of error can take months of debug engineering. That’s the nuance worth naming. The data wasn’t lost to a crash or a failed drive. It was corrupted by a wrong answer, and the loss surfaced downstream long after the bad math.
Why it was silent
ECC and parity protect the large SRAM arrays: caches and register files. The functional datapaths (the integer and floating-point units doing the arithmetic) carry far less protection. A defect there produces a wrong result that looks like a perfectly valid number. Meta discovered these failures were reproducible and data-dependent, not random soft errors: the right operands on the wrong core, every time. Across hundreds of thousands of machines, Meta's screening found hundreds of defective CPUs, at rates that were orders of magnitude higher than soft-error failure-in-time (FIT) models predicted.
Why this matters to AI infrastructure
GPUs and accelerators are not exempt. In Meta's Llama 3 training run, silent data corruption had its own line in the table of unexpected interruptions. Think about what one bad matrix multiply does in data-parallel training. One GPU computes a wrong gradient. The all-reduce averages it into every replica's update in the same step. The error now lives in every copy of the model, and in the next checkpoint. If you catch it, you must roll back hours of work across thousands of GPUs. If you don't, the customer ships a model that is slightly worse for reasons nobody will ever trace. On the inference side, a mercurial core doesn't crash. It serves the wrong tokens to a customer with full confidence.
The discipline of screening from factory to fleet
Silent errors start as tiny silicon defects. Data, temperature, and age make them worse. So screen for them under the conditions where they show up.
- Margin every platform across voltage, frequency, and temperature corners, not just at nominal. A marginal timing path that passes at room temperature and nominal voltage can fail at minimum voltage (Vmin) and high temperature.
- Build the silicon partner's health and screening tools into system-level and rack-level test recipes, then run them at temperature, not on a cold bench.
- Use deterministic workloads with golden reference results, compared bit for bit across devices, so a wrong answer has something to contradict.
- Keep screening after deployment. Meta runs both out-of-production scans and in-production tests across its fleet because some defects show up with age.
Meta's engineers did the whole industry a favor by publishing this. Their findings helped CPU vendors add similar checks to their own tools, and in 2024 the Open Compute Project brought AMD, Arm, Google, Intel, Meta, Microsoft, and So NVIDIA together to solve the problem.
2. The neighbor who knocked
Lifecycle phase: pre-design and margin
What happened
In 2014, researchers proved that you could flip bits in DRAM without ever accessing them. Activate the same row, the aggressor, repeatedly within one refresh window, and the electrical disturbance drains charge from cells in the neighboring victim rows until bits flip. In testing, 110 of 129 DDR3 modules from the three major DRAM vendors were vulnerable. That’s more than 80%. The industry named this anomaly RowHammer.
Why margin caused it
As memory cells shrank and became packed closer together, the room between one cell and its neighbor got thinner. Each process shrink made the problem easier to trigger, not harder. Mitigations such as target row refresh went into DRAM devices, and later research showed some of them could be bypassed. The cat-and-mouse game continued in every new memory generation.
And memory errors weren't rare to begin with. A Google field study found that more than 8% of DIMMs experienced errors every year.
Why this matters to AI infrastructure
HBM is DRAM stacked high and sitting millimeters from a die sizzling at 700 W. In the Llama 3 run, HBM3 memory accounted for 72 of 419 unexpected interruptions (17.2%), second only to faulty GPUs. Heat makes it harder; DRAM cells leak faster when hot, and DRAM standards demand doubling the refresh rate above 85°C, which costs bandwidth. You’ll spend margin you'll never get back if your thermal design runs memory that close to the edge.
For the customer, an uncorrectable memory error usually pulls the GPU out of the job. And, as the Llama 3 team noted, in synchronous training a single GPU failure may require restarting the entire job. A storm of correctable errors can trigger row remapping or page retirement, pushing a GPU into a reset. The worst case is a multi-bit pattern that ECC can't see at all.
The discipline of pre-design and margin
- Start with the use case. Where and how will the product run? Air or liquid? What inlet temperature? What altitude? For how many years? Under which workload: sustained training at full power, or bursty inference? Write the validation plan from these product requirements on day one, not after the boards come back.
- Design to the target with the right margin. Every requirement has to trace to the design that implements it and to a test that proves it. We see micro-examples of this design all the time. Firmware expects five seconds of power hold-up to flush logs and state during an AC loss, and the hardware never implements it. Both teams do their jobs. The requirement just never lives in one place they both own.
- Validate the margin on purpose. Run memory and I/O margin tools: rank margining on DDR, eye margining on PCIe, and high-speed SerDes links. Then inject errors to prove the reliability features work end to end: ECC correction, page retirement, repair of a bad memory location, and the BMC logging chain that tells operators what happened. Prove it before a customer finds out.
3. The rack that ran hot
Lifecycle phase: design. This story is a composite.
What happened
A new liquid-cooled GPU rack went live, serving inference for a customer. Under sustained production load, a board overheated and failed in the data hall. Engineers have a polite name for this: a thermal event. Nobody on the call feels polite about it.
Root cause
The processors were well cooled. The trouble was in the power delivery on the board. To close the layout on schedule, a high-current path was routed narrower than the silicon partner’s design guide allowed. The DC power-integrity analysis, which checks IR drop (the voltage droop across the copper) and current density, would have shown the hot spot. It was not rerun on the final revision. The design review signed off without documented exit criteria, so nobody was required to ask. Under sustained load, I²R heating, which copper turns into heat, did the rest. Resistance rose, the hot spot got hotter, and the board crashed.
What it cost the customer
Every token that board was serving stopped. When a model is sharded across a scale-up domain, losing one board can take down the whole domain, so the blast radius becomes the rack, not the board. Then comes the second wave. Every rack built from the same layout carries the flaw, and the customer waits on capacity they were depending on.
The discipline of design
- Follow the partner's platform design guides for electrical, mechanical, and thermal specs. Treat each rule as a requirement with an owner. If you must deviate, document the deviation and get the partner's sign-off.
- Before the first board is built, simulate signal integrity, power integrity, and thermals. That means checking DC IR drop and current density on every high-current path and modeling the actual cooling setup: liquid plus whatever residual air is left.
- Apply derating rules so every component runs inside its voltage, current, and temperature limits with room to spare, and confirm the results with a full component stress analysis.
- On EVT boards, run maximum-power workloads and take thermal images of the power delivery network. GPU telemetry alone won't show you a hot plane.
- Hold design reviews at set gates throughout the cycle, each with documented exit criteria and a named owner.
- Reuse proven building blocks such as backplanes, power distribution boards, and expanders. A block that has already run on several platforms is one you understand, and it gets you to market faster.
4. The one-in-nine-billion curse
Lifecycle phase: validation
What happened
In late 1994, researchers discovered that Intel's Pentium processor returned slightly wrong answers to some floating-point divisions. The divider used a radix-4 SRT algorithm that looked up quotient digits in a table of 1,066 entries. Five of those entries were missing. For certain operand pairs, the result deviated as early as the fourth significant digit. The classic test case: 4,195,835 divided by 3,145,727 should return 1.333820. The flawed chip returned 1.333739.
Intel said the error would appear once in every nine billion random divisions. Replacing the chips cost $475 million.
Why this matters to AI infrastructure
Rare is not the same as safe. Here's a back-of-the-envelope number: if a petaflop-class accelerator had a flaw at that same rate in a common operation, it would produce 100,000 wrong results every second on every GPU. And our numerics are getting more aggressive, not less: BF16, FP8, and now 4-bit formats; new tensor-core datapaths; and new math libraries in nearly every software drop. Each advancement is new surface for a corner case. Training customers view these bugs as divergence, or as a model that underperforms inexplicably. Inference customers receive answers that are slightly, quietly wrong.
The discipline of validation
Validate in phases and know what each phase is for.
- Engineering validation (EVT) asks whether the board works as designed through power-up and sequencing, rail measurements, electrical validation and hardware verification of every interface, signal integrity, eye margins, and I/O memory margin tools.
- Design validation (DVT) verifies that the system works at the exact intersection of hardware, firmware, BMC, and software. DVT tests performance, thermals at maximum inlet temperature, power, shock and vibration, and regulatory compliance.
- Production validation (PVT) runs at rack and pod scale, replicating the customer’s real workload as closely as possible. This phase tests collective communication across the scale-up and scale-out fabrics, as well as sustained training and inference tools. If the real workload isn't available, we build a synthetic one.
- Numerics deserve their own test plan. Compare the math paths customers actually use against a golden reference, at higher precision or on different hardware.
Moving fast with partners means testing on early silicon and unfinished firmware. That tradeoff is real, so make it purposefully. Write down what you did not test and close the ticket before ramp.
5. The capacitors that waited
Lifecycle phase: derating and ramp
What happened
In 2002, PC motherboard capacitors began to bulge, leak, and fail months after shipping. IEEE Spectrum traced the root cause to a faulty electrolyte, but heat finished the job. An aluminum electrolytic capacitor's lifespan roughly halves with every 10°C rise in ambient temperature.
Run the numbers on a part rated for 2,000 hours at 105°C. At 65°C, that simple rule yields about 32,000 hours of use, roughly 3.7 years. Place the same part next to a hot voltage regulator at 85°C, and lifespan drops to about 8,000 hours, less than one year of 24-7 operation. Ripple-current self-heating makes things worse.
The damage was eye-watering. One PC maker alone shipped more than 11 million potentially faulty desktops between 2003 and 2005, according to court documents.
The boards left the factory healthy. Then they waited.
That's the scary part for anyone running a factory. Those boards passed their tests. First-pass yield can measure today's failures, but it can't catch a part's gradual wear over time.
Why this matters for AI infrastructure
A modern GPU board features dense, high-capacitance power delivery to handle large, rapid load steps as workloads shift between compute and communication phases. The ripple current generates internal capacitor heat. Since these components share identical manufacturing lots and thermal profiles, wear-out is not random; they fail together. For a customer, this means losing nodes in waves months after acceptance and already planning capacity. Now you have mass RMA, cluster downtime, and a hard conversation about trust.
The discipline of derating and ramp
- Everything starts in design. Follow strict derating guidelines, verify each component through a full stress analysis, and keep heat-sensitive parts away from the hottest spots on the board. Calculate capacitor life at the real board temperature and ripple current, against the customer's real service life.
- The ramp phase determines whether a design can be replicated thousands of times using parts from a real supply chain. Build system-level and rack-level test recipes, then run the silicon partner's own health tools inside them.
- Watch first-pass yield. Don't move to mass production until the yield target is reached and holds steady. Stress early builds with extended burn-in and power cycling so weak parts fail on your bench instead of in a customer's hall.
- Every failure category gets a root cause and a formal 8D (eight-discipline) corrective action. Keep lot traceability on critical components, so when a bad lot shows up, you can find every affected board in days, not months. There are no shortcuts here.
Loud failures, silent failures, and the people in between
Large training clusters are built to absorb failure. When Meta trained Llama 3 on 16,384 GPUs, they logged 419 unexpected interruptions in 54 days, about one every three hours. Approximately 78% of the interruptions were attributed to confirmed or suspected hardware issues. GPU issues alone accounted for 58.7%. Effective training time still remained above 90%, and significant manual intervention was needed only three times. Checkpoints, automation, and redundancy carried the load.
Loud failures look like this when you engineer for them. A crash is expensive, but the problem is solvable.
Silent failures are different. A checkpoint can save you from a crash but not from a wrong number. The same is true of an inference answer, a bank ledger, a brokerage account, or a medical record. The systems people rely on have to be right every time. That's where engineering discipline shines.
Behind every discipline in this article is a person. The thermal engineer who reruns the simulation late at night because the inlet temperature in the spec changed. The SI engineer who won't sign off on a marginal eye, even when the schedule is screaming. The test engineer who asks for one more week of burn-in when everyone else wants to ship. I've seen our engineers sleep on the manufacturing floor during a ramp because they wanted to see the next failure with their own eyes. You won't find that in any process document.
At Hyve Solutions we design, validate, build, and deploy under one roof. Our talented engineers in Fremont and Taipei catch problems on a review board or test bench long before a customer would.
And when something does slip through, we write it down. Every bug that reaches the field becomes a lesson in our learning database, and today's lessons become the next generation's design checklists. One engineer's hard night becomes everyone's prevention.
To the younger engineers reading this: if something in a design review makes you uneasy, speak up. The most expensive sentence in hardware is "I thought someone else checked that."
The goal is simple. On Halloween night, the on-call phone stays quiet, the loss curve keeps going down, and the ghost retreats into the mist without ever spooking the customer. The scariest hyperscale hardware stories should be the ones that never happen.
What's the scariest failure you caught before a customer did?
Tell me in the comments. Sanitized versions are welcome.
Rami Khouri is SVP of Engineering at Hyve Solutions, where he leads engineering from design through validation and deployment. He has spent more than two decades building server and rack platforms.
Sources & Further Reading
- Dixit et al., "Silent Data Corruptions at Scale," Meta, February 2021 (arXiv); Meta Engineering blog, "Silent data corruption: Mitigating effects at scale," February 23, 2021 (link); Data Center Knowledge, March 17, 2022 (link)
- Dixit et al., "Detecting Silent Data Corruptions in the Wild," Meta, March 2022 (arXiv)
- Hochschild et al., "Cores That Don't Count," Google, HotOS 2021 (link)
- Open Compute Project, "Computing's Hidden Menace," January 31, 2024 (link)
- Kim et al., "Flipping Bits in Memory Without Accessing Them," ISCA 2014, and the authors' 2023 retrospective (arXiv)
- Schroeder, Pinheiro and Weber, "DRAM Errors in the Wild," Google, SIGMETRICS 2009 (link)
- Meta, "The Llama 3 Herd of Models," July 2024, Section 3.3.4 and Table 5 (arXiv); DCD, July 29, 2024 (link)
- Intel Corp., Form 8-K, January 17, 1995 (SEC)
- Edelman, "The Mathematics of the Pentium Division Bug," SIAM Review, 1997 (link)
- IEEE Spectrum, February 1, 2003 (link)
- Würth Elektronik, "Expected Lifetime of Aluminum Electrolytic and Aluminum Polymer Capacitors," SN008a, June 2018 (PDF)
- IEEE Spectrum, "Dell Tried to Hide Bad Capacitors Problem," June 29, 2010 (link)
