Mike's Notes
Useful source for inference benchmarking. For future reference.
The first part of the article is copied below. You can read the rest by following the link in the resources below.
Resources
- https://inferencex.semianalysis.com/
- https://github.com/SemiAnalysisAI/InferenceX
- https://newsletter.semianalysis.com/p/agentx-inferencexv3-does-cuda-moat
- https://inferencex.semianalysis.com/quotes
- https://newsletter.semianalysis.com/p/claude-code-is-the-inflection-point
References
- Reference
Repository
- Home > Ajabbi Research > Library > Subscriptions > SemiAnalysis
- Home > Handbook >
Last Updated
27/08/2026
Compare Realistic Agentic Inference Perf
.
Since the Claude Code inflection point in November 2025, long-context, multi-turn agentic workloads have grown rapidly. They now dominate traffic for production inferencing. In April 2026, OpenAI’s Enterprise agentic spending overtook ChatGPT spending.
Agentic workflows have decisively taken the baton. Today, we announce AgentX 1.0 - the world’s first fully open source, multi-turn agentic coding inference benchmark at 1 million context, released under Apache 2.0. Our full dashboard is available here.
Source: SemiAnalysis
In the past most measured performance based on fixed sequence length prefill and decode workloads, but this is an inaccurate way to measure workloads. Reality is multi-turn, long context, high prefill reuse, with sub agent bursts, KVCache offload, and numerous tool calls. As such we aimed to build the correct way for the industry to measure AI hardware and software performance.
We have spent more than $3M building this dataset. Today, we open source everything. InferenceXv3 implements AgentX, a new realistic scenario in addition to the existing “fixed sequence length” scenarios (8k1k, 1k1k, 1k8k). It improves the benchmark scenarios by using agentic coding traffic instead of the previous single-turn traffic of 8k input and 1k output tokens.
The full matrix runs on ~2MW of continuously operated compute across over 1000 chips spanning a wide range of SKUs, featuring the MI355X, GB300 NVL72, GB200 NVL72, B300, B200, MI325, MI300X, H200, and RTX Pro Servers. Rubin arrives later this month, and TPUs and Mi455X UALoE72 arrive later this year. Please drop a star if you found our free open source work valuable.
It is great to see amazing performance from both NVIDIA and AMD on agentic workloads. NVIDIA does very good on a lot of frontier models while AMD also does well on some frontier models for specific comparsions.
Source: SemiAnalysis GitHub
The most valuable thing AgentX produced in its first months was not the initial results. It was the massive industry impact the benchmark is already having. Over 70+ upstream PRs for optimizing real world production agentic workloads across vLLM, SGLang, TensorRT-LLM, ATOM, AITER, Dynamo, LMCache, and Mooncake, uses AgentX as the north star benchmark proxy. Most of these optimization improvements are transferable to production traffic. We deep dive into each of these optimizations later in the article.
Open source is a core principle for InferenceX and thus, we open more of the stack than most people who use that word. That includes an open frontend, a public database served through an easily consumable REST API that multiple tier 1 AI lab’s capacity planning teams already consume, public GitHub Actions CI provenance, logs, and accuracy validation on every single point. Crucially, our benchmark configs mainly track recipes.vllm.ai and SGLang cookbook on upstream images such that we are measuring the performance actual customers are experiencing instead of measuring benchmax’ed images.
In three to four weeks, we will release an AgentX update article. It will cover further optimizations to agentic workloads, plus updated performance results from AMD and Nvidia. It is important to understand that the profile of agentic workloads is updating fast. InferenceX will continue to move swiftly to benchmark the relevant workloads.
InferenceX is 100% committed to being open-source - this would not be possible without the contributions and support from our OSS partners. We would like to thank the following people that have made massive contributions to the AgentX 1.0 release:
- Inferact/vLLM: Roger Wang, Yifan Qiao, Simon Mo, Jeff Ma, and many others
- RedHat/llm-d: Michael Goin, Robert Shaw, Tyler Michael Smith
- RadixArk/SGLang: Baizhou Zhang, Yuwei An, Mingyi Lu, and many others
- LMCache/TensorMesh: Samuel Shen
- Weka: Callan Fox, ValB
- MoonCake Maintainers: Teng Ma, Xu Wenjie, Ke Yang
- AMD: Thomas Wang, HaiShaw, Andy Luo, Seungrok Jung, Chun Fang, Parth Panchal, Bill He, Theresa Shan, Hongxia, Fangzhou, Gilbert Lei, Yanfei Wang, Duyi Wang, Peng Sun, Lingpeng Jin, Simon Danielsson, Xiaohu Guo, Haichen Zhang, Chang Liu, Doug Lehr, Poovaiah Palangappa, and many others in the AMD Shanghai Development Centre
- Nvidia: Xin Li, Anthony Casagrande, Kedar Potdar, Ankur Singh, Ishani Dhanani, Nick Comly, Nvidia Shanghai TensorRT-LLM team, and many others
- Anthropic staff, for promptly fixing multiple bugs that made implementing AgentX possible
- GitHub: Austen Stone for helping with reliability of GitHub Actions that AgentX uses
- And many others
Source: InferenceX
... See the original article for the rest.
No comments:
Post a Comment