Jeremy Blankenship
infrastructure engineer · oklahoma city
Hi, I'm Jeremy. I've worked in enterprise server and AI infrastructure since 2010. Lately that has meant helping stand up some of the largest AI clusters in the world, and the work that comes after go-live: figuring out why a GPU system won't come up, why the same failure keeps coming back, and what has to change so it stops.
The part I like best is the root cause. Swapping a part makes the alert go away. I want to know why it broke, how to prove it, and what keeps it from happening again on the next hundred nodes. I also like a good disagreement, the kind where both people walk away understanding the problem better. "I don't know yet" is a fine answer. A confident guess isn't.
At home I run a lab that has gotten a little out of hand: Proxmox, OPNsense, ZFS, more monitoring than a house needs, a 32 GB Tesla V100 in a desktop that wasn't built for one, and two 128 GB NVIDIA GB10 machines cabled together because one wasn't enough. GB10 (sm_121) is close enough to other Blackwell chips that vendor kernels try to run on it, and different enough that some of them break. When FlashInfer's FP4 kernels kept crashing it with Xid 13 faults, GPU core dumps traced the crash to a TMA load through a null descriptor. I patched FlashInfer, CUTLASS, and vLLM locally until day-long serving runs logged no new Xids.
lab notes
- The GPU that wasn't there. I added a 32 GB Tesla V100 to a desktop.
lspcicould see it;nvidia-smicouldn't. Two causes were stacked: the open kernel module can't drive Volta because the chip has no GSP, and the current proprietary driver branch had dropped Volta entirely. Moving both cards to the 580 legacy driver fixed it, which then made the desktop flicker until a udev rule told the compositor to ignore the headless card. Being on the bus isn't the same as being supported. - The network works. So what? Two GB10s serving one model over 200 Gb/s RoCE. A NCCL protocol tweak made the all-reduce microbenchmark about 20% faster and real serving 0.4% slower. The model does 87 tiny all-reduces per generated token, and inside real serving each one took around ten times longer than in isolation. A passing benchmark doesn't prove the workload will scale.
- The twelve-minute prompt. A cold 262K-token prefill on the V100 took 11.7 minutes. Splitting attention across the head dimension cut that to 9.8, about 19% faster. Before keeping it, I checked outputs against an FP64 reference and confirmed the generated tokens matched the old path at 32K, 128K, and 258K context.
- More speculation, less speed. On a held-out test set, drafting four or five tokens per step instead of three looked a few percent faster. Replayed against real tool-calling traffic, it was 10 to 12% slower, because real requests didn't accept enough drafts to pay for the extra verification. Three stayed.
things i've built
- Enoch: a control plane for unattended AI research runs on a single GPU. It handles queueing, dispatch gates, process and GPU telemetry, and an audit gate that won't publish a result it can't tie back to evidence. Archived.
- CouncilRouter: asks several LLMs the same question, has them critique each other, and keeps a devil's advocate in the room. It treats consensus as a signal, not proof.
- Small plugins for the Hermes agent: memory search over MCP, webhook filtering, and tool-schema trimming.
- A steady stream of model-architecture experiments: hybrid SSM/attention, quantization paper reproductions, initialization tricks. Most of them don't work. That's the fun part.
misc
- I came back to Astonia after almost twenty years and found I'd missed the people more than the game.
- I rented a game server to play on and ended up designing an autonomous moderation system for it instead.
- I've 3D-printed custom cooling ducts for computers that already had fans.
- I will benchmark two nearly identical configurations just to understand a 2% difference.
- Away from the keyboard: fishing and canoe camping. Neither needs a terminal, which is part of the appeal.
- I'm teaching myself the harmonica (a ten-hole in C) and how to sing. Not at the same time.
- I have seriously considered getting into O-gauge model trains. Nobody has talked me out of it yet.
- Genealogy is just root-cause analysis with older logs.
- Favorite movies: The Count of Monte Cristo (2002), Law Abiding Citizen, The Equalizer. Also anything with superheroes or a rogue AI.
- I'll watch any Adam Savage build, then go find a reason to open CAD.