Nvidia team tells that large context windows are frequently overkill
In many tests, summaries and lookups gave about the same result as a large context window.
A Nvidia team tells that large context windows of language models are frequently “overkill” and have a large cost. A context window is the memory of a model: all that it holds in view when it writes a reply. The team tells that in many tests, periodic summaries and targeted lookups gave about the same result.
How it works
A system makes summaries at set times. It also does lookups, which fetch only the parts that a task uses.
The numbers
The part of the report that we have gives no test results. It gives only the growth of windows. GPT-4 launched in March 2023 with 8,192 tokens, which are small units of text. GPT 5.6 Sol handles around 1 million tokens, about 120 times as much.
What is not known
The report gives no names of tests, no data on the models and no numbers for cost. The team tells “in many tests”, not always.
Insight: Vendors sell more context while Nvidia questions the need
In October 2026, OpenAI increased a default context from 272k to 500k. In November 2026, Google added training on inputs of 14 million tokens. Those moves push for more context, but the team above tells that it is frequently overkill. Cost for each task possibly becomes the test.
Sources
Posted