In my previous blog, we explored the enormous amount of energy consumed in training Large Language Models (LLMs). That was just the beginning of the story. Once trained, these models move to the next crucial phase: inference. This is when the model, now armed with vast knowledge, is finally put to work. Every time you ask ChatGPT a question, request a poem, or use an AI assistant to write an email or translate a document, the model is performing inference. In simpler terms, inference is the “use phase” of the model i.e. when it takes in new input and produces output, like responding to you right now.
Before diving into how much energy this process consumes, let’s appreciate where LLM inference is already making a big difference. Think about intelligent chatbots that can help you book a flight, content generation tools that draft blog posts/ create cinematic clips or business emails in seconds, and even virtual tutors that assist with math problems. Many people have started using ChatGPT or Gemini as search engines, replacing the need to browse dozens of websites. These aren’t just cool gimmicks; they are real productivity tools. And they are growing fast. With around 5 billion of interactions every month, LLMs are becoming a backbone of modern digital life.
But here comes the twist. While training a model like GPT-4 is energy-heavy, inference happens far more frequently. It’s repeated millions of times a day. That adds up. In fact, according to a report from Google, inference now accounts for nearly 60% of total energy use in their AI workloads. Meta revealed something even more striking: within their AI infrastructure, power is distributed in a 10:20:70 ratio among experimentation, training, and inference respectively, with inference taking the lion’s share.
Let’s put this into perspective. A single interaction with ChatGPT can consume around 2.9 watt-hours (Wh) of electricity. Compare this to a traditional Google search, which only uses about 0.3 Wh – ChatGPT uses nearly a ten times more energy per query. Integrating ChatGPT into the 9 billion daily Google searches could raise global electricity demand by approximately 10 terawatt-hours annually, equivalent to about 1/60 of Canada’s total annual electricity consumption.
Now, the question is what mostly impacts energy consumption during this inference process? The answer lies in several factors. First, larger models, like GPT-4, are more accurate and helpful but demand significantly more power. The complexity and length of input and output also heavily influence consumption. Simple yes-or-no questions require fewer computations than detailed, multi-step explanations. Even a single extra word matters. OpenAI CEO Sam Altman recently highlighted that polite phrases like “please” and “thank you” add millions of dollars to operational costs, as each additional word increases computational demand and thus energy use. If the model is generating images or videos instead of just text, the energy demand increases even further.
The hardware used and how it’s configured can either reduce or amplify these costs. Graphical Processing Units (GPUs), for example, offer great parallel processing power but consume a lot of electricity. Efficient batching (grouping several queries together for simultaneous processing) can help, as can techniques like GPU power capping. Still, these are stopgaps rather than solutions.
There are also behavioral and societal trends to consider. The better and faster a model becomes, the more people use it. Ironically, the efficiency of LLMs can backfire due to what’s known as the rebound effect: more efficiency encourages more use, leading to more total consumption. And the trend towards ever-larger models only intensifies this issue.
So, what can we do? First, we need more transparency. Companies like Meta have started sharing their models’ energy and carbon usage, but more openness from the big techs such as OpenAI and Google is crucial. We also need standardized protocols, such as AI energy scores, that help compare models based on efficiency, just like Energy Star ratings for appliances. Developers can be encouraged to pick the right-sized model for a task rather than defaulting to the largest one. Architecture like Mixture-of-Experts (MoE), which activate only a fraction of the model during inference, can preserve performance while drastically cutting energy use (e.g., Deepseek-R1).
On the hardware side, there’s hope. Advances in chip design are enabling faster and more energy-efficient AI processors. And Edge AI, running models closer to the data source like on smartphones or local servers, can reduce the need to shuttle data back and forth from big data centers, cutting down on energy and latency.
Even on the policy level, there’s progress. The recent summits on the Future of Energy Security in London and AI Action Summit in France recognized AI’s potential not just as a problem, but also as a solution to improve forecasting and efficiency in the energy sector itself.
Inference may seem invisible to the average user: a tap of a button, a few seconds of response. But behind that simplicity lies a vast, complex web of computation that quietly burns through megawatts of power. The good news is, we’re starting to understand it better. And understanding is the first step toward change. As we observe Canadian Environment Week from June 2 to 8, 2025, it’s an opportunity for us to reflect on the environmental impacts of our digital habits, including the energy consumption of AI tools we use daily.
Photo by Google DeepMind
