Citi: AI Inference Demand Remains Tight, Bottleneck Shifting From Chips to Power and Data Centers

June 16, Citigroup noted that demand for AI inference remains robust, with a shortage of computing power spreading from the latest generation of chips to prior GPUs, forcing model vendors to accelerate monetization through tactics like dynamic pricing, usage quotas, and routing mechanisms. In a June 14 report, analysts including Heath Terry found that rental rates for A100 GPUs rose 0.6% over the past week, and 11% cumulatively over six weeks—signaling AI computing demand isn’t limited to only cutting-edge hardware. The firm added that some state-of-the-art models have lifted prices sharply after improving their intelligence benchmarks, with Citigroup observing that “scarcity is being monetized faster than it’s being solved.” The report also highlighted that no model vendor currently boasts all three key advantages: intelligence, speed, and affordability. Top-tier models have seen their intelligence scores climb roughly 4 points, yet their overall prices have nearly doubled. Meanwhile, mid-range models have made gains in speed, with the median output rate of the top 20 models jumping from 64 tokens per second to 105 tokens/s over six weeks. Additionally, the gap in capabilities between closed-source and open-source models is widening: proprietary models now lead open-source ones by around 10 points in intelligence scores, up from approximately 6 points previously. This has led top model providers to prioritize locking in the high-end market with superior performance, rather than competing on price against open-source alternatives. Beyond computing power, electricity costs and data center location are emerging as new constraints on AI expansion. The firm cited a private neocloud that has signed contracts for 4.9GW of demand, though its planned pipeline exceeds 40GW—revealing a massive chasm between surging demand and available supply. Data centers tend to be sited in regions with electricity prices of 9–12 cents per kilowatt-hour, while the share of renewable energy and long-term power purchase agreements also influence location choices. Looking ahead, Citigroup projects AI infrastructure costs will continue to climb. Rising component prices, power access fees, and upfront infrastructure investments are driving higher capital expenditure (capex) for equivalent compute power—such as H100 GPUs—with electricity costs shifting from operational expenses to upfront capital outlays. The next wave of value may flow to the “inference routing layer”: a platform that automates choosing the right model, quantization method, and hardware for individual tasks to lower inference costs and boost efficiency. However, implementation faces hurdles like enterprise data security, intellectual property concerns, and privacy protection. Across the broader AI ecosystem, the report notes growth isn’t limited to GPUs alone—it extends to data centers, power, optical communications, cloud infrastructure, and model applications. Citigroup named relevant coverage targets in its appendix: Ciena, Lumentum, and MiniMax, demonstrating that the AI inference cycle is expanding from chips to a wider range of infrastructure and application segments.