Groq Sets New AI Inference Speed Record at 1,200 Tokens Per Second
Groq has set a new AI inference speed record, achieving 1,200 tokens per second on its second-generation LPU (Language Processing Unit) hardware running the Llama 4 70B model. This is 3x faster than Groq's previous record and approximately 30x faster than typical GPU-based inference. The speed breakthrough is enabled by Groq's new GroqRack Gen2 system, which uses an improved memory architecture and compiler optimizations to eliminate the memory-bandwidth bottleneck that limits GPU inference. For end users, the improvement means AI responses that stream as fast as reading speed, with time-to-first-token under 50 milliseconds — faster than human perception. Groq reports that its platform now serves over 500 million API calls per month, up from 100 million six months ago, driven primarily by applications requiring real-time AI interaction. Key customers include voice AI platforms that need sub-200ms response times, interactive coding assistants, and real-time translation services. The company also announced partnerships with two major telecommunications companies to deploy LPU nodes at edge locations, bringing ultra-fast inference closer to end users. Groq's free tier remains available with rate limits, while production customers pay a premium over GPU-based providers in exchange for the speed advantage.
Groq has set a new AI inference speed record, achieving 1,200 tokens per second on its second-generation LPU (Language Processing Unit) hardware running the Llama 4 70B model. This is 3x faster than Groq's previous record and approximately 30x faster than typical GPU-based inference.
The speed breakthrough is enabled by Groq's new GroqRack Gen2 system, which uses an improved memory architecture and compiler optimizations to eliminate the memory-bandwidth bottleneck that limits GPU inference.
For end users, the improvement means AI responses that stream as fast as reading speed, with time-to-first-token under 50 milliseconds — faster than human perception.
Groq reports that its platform now serves over 500 million API calls per month, up from 100 million six months ago, driven primarily by applications requiring real-time AI interaction.
Key customers include voice AI platforms that need sub-200ms response times, interactive coding assistants, and real-time translation services.
The company also announced partnerships with two major telecommunications companies to deploy LPU nodes at edge locations, bringing ultra-fast inference closer to end users.
Groq's free tier remains available with rate limits, while production customers pay a premium over GPU-based providers in exchange for the speed advantage.
The company closed a $640 million Series D at a $5.8 billion valuation, with the funding directed toward manufacturing additional LPU systems. CEO Jonathan Ross noted that inference compute demand is growing faster than training compute demand, validating Groq's inference-first approach to AI hardware.
Related Tools
More News
NVIDIA Launches NIM Microservices for Enterprise AI Deployment
NVIDIA has launched NIM (NVIDIA Inference Microservices), a suite of containerized AI model serving packages that reduce enterprise AI deployment time from weeks to hours with optimized inference performance.
AI Agents Market Reaches $15 Billion as Enterprise Adoption Surges
The global market for AI agents — autonomous AI systems that can plan, execute, and iterate on complex multi-step tasks — has reached $15 billion in annual spending, according to a new report from McKinsey. This represents a 200% increase from 2025, driven by enterprise adoption of agentic AI for customer service, software development, data analysis, and business process automation. The report identifies three tiers of AI agent adoption: basic agents that handle single-step tasks like email responses and appointment scheduling (adopted by 65% of enterprises), intermediate agents that manage multi-step workflows like report generation and data pipeline management (35% adoption), and advanced agents that autonomously execute complex processes like code deployment and financial analysis (8% adoption). The largest spending categories are customer service agents ($4.2B), coding agents ($3.8B), and data analysis agents ($2.5B). McKinsey projects the market will reach $45 billion by 2028 as agent reliability improves and enterprises become more comfortable delegating complex decisions to AI. Key enabling platforms include OpenAI's Agents SDK, Anthropic's Claude computer-use capabilities, and LangChain's agent framework. The report warns that agent governance and monitoring remain underdeveloped, with most enterprises lacking adequate oversight mechanisms for autonomous AI actions.
Microsoft 365 Copilot Gets Custom AI Agents and Actions
Microsoft has updated 365 Copilot with custom AI agent creation, allowing organizations to build agents that automate complex workflows spanning Word, Excel, Outlook, Teams, and SharePoint without code.
GPT-5.2's Agentic Mode Transforms Enterprise Workflows
OpenAI's GPT-5.2 introduced a fundamentally new approach to agentic task completion that is already transforming enterprise workflows. The model can now maintain coherent plans across 50+ sequential tool calls with parallel execution, reducing latency in complex automation pipelines by up to 60%. Early enterprise adopters report that GPT-5.2's agentic mode handles tasks like multi-step data analysis, cross-platform content publishing, and automated code review workflows that previously required custom orchestration code. The key innovation is what OpenAI calls deliberative alignment — a training approach that lets the model dynamically allocate compute to harder sub-tasks while breezing through simpler ones. This means a single agentic session can handle both quick lookups and deep reasoning without manual configuration. Several Fortune 500 companies have reported 40-70% time savings on analyst workflows by deploying GPT-5.2 agents through the API. However, reliability remains a concern — OpenAI acknowledges a 3-5% failure rate on chains exceeding 30 steps, and enterprise deployments require human-in-the-loop checkpoints for critical decisions.