Benchmarking 15 Old Tesla GPUs for Modern AI Inference and Training

The Experiment: 15 Teslas vs. Modern AI

A hardware hacker dropped a gauntlet: take 15 ancient NVIDIA Tesla GPUs—cards that hyperscalers retired years ago—and force them through modern AI inference and training loops. We’re talking Kepler, Maxwell, and early Pascal silicon. The K80, M40, P4, P40, and P100. Cards that cost between $30 and $150 on eBay. The goal wasn’t nostalgia. It was a cold, quantitative answer to a question every tinkerer asks: Can I serve LLaMA from literal trash?

The test suite ran Llama 2 7B, Mistral 7B, and a handful of smaller models through both inference (tokens-per-second) and fine-tuning (LoRA on Alpaca). Power draw was measured at the wall. The results are a masterclass in how architectural bottlenecks age—and where they don’t.

Why This Matters for the Working Engineer

If you’re an engineer, especially an FDE (Forward Deployed Engineer) stitching together prototypes for customers, this isn’t academic. You’re constantly trading off latency, throughput, and cost. Cloud GPU instances are expensive. A single A100 on Lambda Labs runs over $1.10/hr. A stack of four P40s in a used Dell server costs less than a month of that rental.

The benchmark answers three practical questions:

1. What’s the cheapest card that can actually run a 7B model at reading speed? (Spoiler: it’s not the one you think.)
2. Does FP16 performance matter if you’re forced to quantize? (Yes, but not for the reason you assume.)
3. Where does training become physically impossible, not just slow?

For the FDE building a local RAG pipeline or an on-premise support agent—like the one we detailed in our WhatsApp Customer Support Agent guide—knowing the floor is critical. You don’t need an H100 to run a 4-bit quantized Mistral. You might need nothing more than a card that costs less than the power supply driving it.

The Raw Numbers: VRAM is King, Bandwidth is Queen

Let’s get the headline figures on the table. These numbers are for single-batch inference on Llama 2 7B (4-bit GPTQ quantization) unless noted.

Data synthesized from the original benchmark. Tokens/sec measured with llama.cpp, 2048 context.

The lesson is immediate: VRAM capacity is the hard gate. The P4 is a marvel of efficiency at 50W, but 8GB locks it out of 7B models entirely. It’s a 3B card. The K80 has 24GB but splits it across two physical GPUs with a slow bridge, so effective bandwidth to any single model is halved. The P40 is the sweet spot: 24GB, reasonable bandwidth, and a price that’s almost a rounding error.

But look at the P100. Same price as a P40, half the VRAM, but nearly double the tokens-per-second. That’s the 732 GB/s HBM2 bandwidth screaming. For models that fit in 16GB—Mistral 7B at 4-bit fits comfortably—the P100 is a rocket. It’s the card you buy when you’ve sworn off 70B models forever.

Architectural Decay: Why Old Compute Dies First

Here’s the nuance the raw t/s numbers hide. The benchmark also tested FP16 training throughput for a small LoRA fine-tune. This is where the generational gaps become chasms.

The K80 and M40 are based on Kepler and Maxwell architectures. They lack hardware-accelerated FP16. When PyTorch requests half-precision, these cards silently promote to FP32, do the math, then cast back. You get none of the memory savings and a compute penalty. The P40 (Pascal) has FP16 support, but it’s slow—roughly 1/64th the rate of FP32, a deliberate segmentation move by NVIDIA at the time. The P100 has proper, fast FP16.

In practice, a LoRA fine-tune on the P100 completes in a time that’s annoying but workable. On the P40, it’s an overnight job. On the K80, it’s a calendar event. The benchmark author noted that training on the K80 was so slow that the electricity cost to complete a single epoch exceeded the purchase price of the card.

This is the “capability cliff.” Inference is a bandwidth problem. Training is a compute problem. Old GPUs have enough bandwidth to shuffle weights from VRAM to registers for inference. They do not have the tensor cores or fast FP16 pathways to do the backward pass efficiently. If you’re prototyping a fine-tuning pipeline, the P100 is the absolute floor. Anything older is a lesson in patience you don’t need.

Practical Playbook: Running Inference on E-Waste Today

So you want to build a budget inference node. Here’s the engineer’s checklist, drawn from the benchmark’s methodology and hard-won community knowledge.

1. Pick your card by model size, not by benchmark charts.

• 3B parameters or smaller: Tesla P4. It’s 50W, single-slot, and passively cooled in a server chassis. You can stuff eight of these in a 2U box and serve dozens of small models concurrently.
• 7B parameters: Tesla P40 or P100. P40 if you need 32K context and the extra VRAM headroom. P100 if you need low latency and the model fits in 16GB.
• 13B parameters: You’re in multi-GPU territory. Two P40s with tensor parallelism will work. Two P100s will hit the 16GB wall. The K80 technically has 24GB but the split architecture makes it painful.
• 70B parameters: Don’t. Just don’t. Rent cloud for this.

2. Quantization is not optional.

You will run 4-bit GPTQ or AWQ quants. No exceptions. The benchmark showed that FP16 inference on a 7B model requires 14GB of VRAM minimum, which knocks out the P100 and makes the P40 a single-task card. At 4-bit, that same model uses ~5GB, leaving room for a large KV cache. This is the only reason the K80 is even viable—the quantization fits in each 12GB half.

3. Software stack matters more than drivers.

The original benchmark used llama.cpp with CUDA acceleration. This is the right call. The proprietary NVIDIA drivers for these cards are stuck in legacy branches. The K80 requires the 470.x driver series, which is entering end-of-life. But llama.cpp compiles against the CUDA toolkit directly and doesn’t care about driver versions as long as the kernel module loads. Other options:

• ExLlamaV2: Faster on Pascal and newer. The P40 and P100 see a 10-15% uplift over llama.cpp for GPTQ models. Not worth it for Kepler/Maxwell.
• vLLM: For actual serving with continuous batching. The P40 and P100 can run vLLM, but the K80 and M40 lack the compute capability for the attention kernels vLLM expects. Stick to llama.cpp’s server mode for those.

4. Power and cooling are the real cost.

A P40 draws 180W under load. At $0.12/kWh, running 24/7, that’s $189/year in electricity. The card costs $150. In two years, you’ve paid for it twice in power. Compare that to a modern RTX 4060 Ti 16GB, which draws 100W and delivers 2-3x the tokens-per-second. The TCO math gets ugly fast for always-on workloads.

If you’re an FDE deploying a proof-of-concept for a client, like the Job Application Autofill Agent we built with free LLMs, the calculus changes. You’re not running 24/7. You’re spinning up for demos and batch inference runs. The P40’s idle power is 25W. That’s fine. The capital expense is near zero. The operational expense is a rounding error. This is the exact niche where e-waste GPUs shine: bursty, prototype workloads where the alternative is a $0.50/hr cloud instance that sits idle 90% of the time.

A Balanced Take: The Hidden Costs of Jank

I’m not going to cheerlead this. The benchmark is fascinating, but running production inference on decade-old silicon is a decision with sharp edges.

The compatibility tax is real. The K80 is a Kepler GPU. Compute Capability 3.7. PyTorch dropped Kepler support in version 1.11. You cannot install a modern PyTorch binary and expect it to work. You’ll compile from source or pin ancient versions. The M40 (Maxwell, CC 5.2) is on the same trajectory. The P40 and P100 (CC 6.0 and 6.1) are still supported in PyTorch 2.x, but for how long? NVIDIA’s CUDA deprecation policy is aggressive. The moment CC 6.x support is dropped, your $150 P40 becomes a paperweight for anything except llama.cpp—which, to be fair, might be all you need.

The failure modes are silent. These cards are pulled from decommissioned servers that ran for 5-7 years at 80°C. The thermal paste is dust. The fans—if they have them—are worn. The benchmark author noted that one K80 arrived DOA. Another M40 had a fan bearing that screamed like a banshee. You’re not buying a GPU; you’re buying a project. If your time is worth more than $50/hr, one afternoon of debugging a dead card erases the savings versus buying a used RTX 3060.

The performance ceiling is low and hard. The P40 delivers 15 t/s on a 7B model. That’s just above reading speed for a single user. Add a second concurrent request and you’re at 7 t/s each. Continuous batching helps, but the P40 lacks the tensor core grunt to parallelize attention across multiple sequences efficiently. You will not build a multi-tenant API on this hardware. You’ll build a single-user tool. That’s fine for a personal assistant or a local RAG pipeline, but it’s not a product.

The real alternative isn’t an A100—it’s a used 3090. A used RTX 3090 costs $600-700. It has 24GB of GDDR6X at 936 GB/s. It delivers 80+ t/s on the same 7B model. It’s supported by every framework, every kernel, every cutting-edge quantization method. Over a 3-year lifespan, including electricity, the 3090 is cheaper per token than a P40. The P40 wins only if you need a card right now and have exactly $150. That’s a valid scenario. Just don’t confuse it with a long-term strategy.

This connects directly to the skills we emphasize for FDEs in The Highest-Leverage Skills for an FDE in the AI Era. Knowing the hardware tradeoffs—understanding that a P100’s bandwidth makes it an inference beast but its 16GB cap makes it a training non-starter—is the kind of architectural thinking that separates engineers who ship from engineers who spec. You’re not just picking a GPU. You’re picking the constraints your entire stack will live within.

FAQ

Q: Can I train a LoRA on a Tesla K80? A: Technically yes. Practically, no. The K80 lacks hardware FP16 support, so training throughput is measured in tokens-per-minute, not per-second. A single epoch on Alpaca could take a week. The electricity cost will exceed the card’s value. Use it for inference only.

Q: What’s the best Tesla GPU for a 13B model? A: Two Tesla P40s with tensor parallelism. 24GB each gives you 48GB total, enough for a 13B model at 4-bit with a large context. A single P40 will OOM at 13B even quantized if you push context beyond 2048. The P100’s 16GB is insufficient.

Q: Do these cards support FlashAttention? A: No. FlashAttention requires Compute Capability 7.5 (Turing) or higher for the efficient CUDA kernels. The Tesla P100 is CC 6.0, P40 is 6.1. You’ll use standard attention implementations, which are slower and use more VRAM for the attention matrix. This is another hidden performance penalty.

Q: Should I buy a P40 or a P100 for inference? A: If your model fits in 16GB at your target quantization and context length, buy the P100. It’s 80% faster. If you need 24GB for larger models or long context, buy the P40. The P40 is the more flexible card; the P100 is the faster card within its memory limit.

Q: Can I put these in a normal desktop PC? A: With caveats. Tesla cards have no video outputs and typically use passive cooling designed for server chassis with high-static-pressure fans. In a desktop, you’ll need to either zip-tie a fan to the card or ensure your case has a fan wall directly in front of the GPU area. The P4 is an exception—it’s a low-profile, 50W card that can survive on passive airflow in many cases. Also, Tesla cards use EPS 12V power connectors, not PCIe power. You’ll need adapters.

Q: Is this actually e-waste, or is the term hyperbolic? A: The term is deliberately provocative. These cards are functionally obsolete for the datacenters they were designed for, which have moved to A100s and H100s. But for a specific, narrow use case—single-user, quantized inference on a budget—they’re surprisingly capable. They’re more like “computational second-life hardware.” The real e-waste is the K80 and M40, which are nearing total software deprecation. The P40 and P100 have a few years of useful life left.