About
The story behind Poor Paul's Benchmark and why everything is open.
Project Philosophy
Poor Paul's Benchmark (PPB) was created because real-world AI inference performance data is surprisingly hard to find. Most published benchmarks come from hardware vendors or cloud providers testing under ideal conditions. PPB measures inference on the hardware people actually own — consumer GPUs, workstations, and small-business servers.
The goal is simple: give anyone an honest, reproducible way to measure and compare LLM inference performance across models, quantizations, hardware, and software stacks.
Open Benchmark, Open Data
Everything about PPB is open:
- The CLI is open source under the MIT license. GitHub →
- The raw dataset is public on Hugging Face. Every row is downloadable, filterable, and ready for your own analysis. Dataset →
- The dataset is append-only. Results are never edited or deleted. Deduplication and curation are applied at the view layer, not at the source.
The Ecosystem
PPB is three connected open-source components:
| Component | What it does |
|---|---|
poor-pauls-benchmark | CLI that runs benchmarks and publishes results |
ppb-results dataset | Append-only public ledger on HuggingFace |
ppb-mcp | MCP server — lets LLMs query the dataset directly |
poorpaul.dev is the human-readable layer on top of the same dataset. Query this data with an LLM: connect any MCP client to mcp.poorpaul.dev.
What is poorpaul.dev?
The raw Hugging Face dataset is great for data scientists and researchers. poorpaul.dev exists for everyone else. We provide:
- Curated leaderboards — deduplicated, best-per-combo rankings so you can quickly compare hardware.
- Interactive exploration — charts that let you see how throughput, latency, and context size interact.
- Benchmark writeups — analysis articles that contextualize the numbers and highlight interesting trends.
The site is statically generated from the public dataset and hosted as plain HTML on AWS. No server, no database, no tracking.
Contributing
The quickest way to contribute is to run benchmarks on your own hardware. Install the Poor Paul's Benchmark CLI, run a test, and your results will be published to the Hugging Face dataset automatically. They'll appear on this site after the next data refresh.