Groq Sets New AI Inference Speed Record at 1,200 Tokens Per Second
Groq has achieved a record 1,200 tokens per second inference speed for the Llama 4 70B model using its custom LPU (Language Processing Unit) chips, making it the fastest publicly available inference service.
Groq has set a new public inference speed record, serving the Llama 4 70B model at 1,200 tokens per second on its custom LPU (Language Processing Unit) architecture. This is approximately 4x faster than the fastest GPU-based inference services and opens new possibilities for real-time AI applications.
The speed achievement is enabled by Groq's unique chip architecture, which uses a deterministic execution model rather than the stochastic approach of GPU-based inference. This eliminates memory bandwidth bottlenecks that limit GPU inference speed, allowing Groq's chips to deliver consistent, predictable latency.
At these speeds, AI responses appear virtually instantaneous, enabling new interaction paradigms. Groq demonstrated a real-time coding assistant that suggests code as fast as a developer can read it, a conversational AI with zero perceptible latency, and a document analysis system that processes 100-page documents in seconds.
Groq has expanded its cloud infrastructure to six data centers globally, reducing network latency for users worldwide. The company offers API access starting at $0.27 per million input tokens for Llama 4 70B, competitive with GPU-based providers despite the significant speed advantage.
The company has also announced partnerships with several enterprise customers including Bloomberg, Shopify, and Twilio, who are using Groq's infrastructure for latency-sensitive applications. CEO Jonathan Ross noted that speed is not just a convenience feature but enables entirely new AI application categories that were previously impractical.
Related Tools
More News
AI Agents Market Reaches $15 Billion as Enterprise Adoption Surges
The global market for AI agents — autonomous AI systems that can plan, execute, and iterate on complex multi-step tasks — has reached $15 billion in annual spending, according to a new report from McKinsey. This represents a 200% increase from 2025, driven by enterprise adoption of agentic AI for customer service, software development, data analysis, and business process automation. The report identifies three tiers of AI agent adoption: basic agents that handle single-step tasks like email responses and appointment scheduling (adopted by 65% of enterprises), intermediate agents that manage multi-step workflows like report generation and data pipeline management (35% adoption), and advanced agents that autonomously execute complex processes like code deployment and financial analysis (8% adoption). The largest spending categories are customer service agents ($4.2B), coding agents ($3.8B), and data analysis agents ($2.5B). McKinsey projects the market will reach $45 billion by 2028 as agent reliability improves and enterprises become more comfortable delegating complex decisions to AI. Key enabling platforms include OpenAI's Agents SDK, Anthropic's Claude computer-use capabilities, and LangChain's agent framework. The report warns that agent governance and monitoring remain underdeveloped, with most enterprises lacking adequate oversight mechanisms for autonomous AI actions.
NVIDIA Launches NIM Microservices for Enterprise AI Deployment
NVIDIA has launched NIM (NVIDIA Inference Microservices), a suite of containerized AI model serving packages that reduce enterprise AI deployment time from weeks to hours with optimized inference performance.
Microsoft 365 Copilot Gets Custom AI Agents and Actions
Microsoft has updated 365 Copilot with custom AI agent creation, allowing organizations to build agents that automate complex workflows spanning Word, Excel, Outlook, Teams, and SharePoint without code.
GPT-5.2's Agentic Mode Transforms Enterprise Workflows
OpenAI's GPT-5.2 introduced a fundamentally new approach to agentic task completion that is already transforming enterprise workflows. The model can now maintain coherent plans across 50+ sequential tool calls with parallel execution, reducing latency in complex automation pipelines by up to 60%. Early enterprise adopters report that GPT-5.2's agentic mode handles tasks like multi-step data analysis, cross-platform content publishing, and automated code review workflows that previously required custom orchestration code. The key innovation is what OpenAI calls deliberative alignment — a training approach that lets the model dynamically allocate compute to harder sub-tasks while breezing through simpler ones. This means a single agentic session can handle both quick lookups and deep reasoning without manual configuration. Several Fortune 500 companies have reported 40-70% time savings on analyst workflows by deploying GPT-5.2 agents through the API. However, reliability remains a concern — OpenAI acknowledges a 3-5% failure rate on chains exceeding 30 steps, and enterprise deployments require human-in-the-loop checkpoints for critical decisions.