What matters in AI.

Subscribe

Nvidia team tells that large context windows are frequently overkill

In many tests, summaries and lookups gave about the same result as a large context window.

A Nvidia team tells that large context windows of language models are frequently “overkill” and have a large cost. A context window is the memory of a model: all that it holds in view when it writes a reply. The team tells that in many tests, periodic summaries and targeted lookups gave about the same result.

How it works

A system makes summaries at set times. It also does lookups, which fetch only the parts that a task uses.

The numbers

The part of the report that we have gives no test results. It gives only the growth of windows. GPT-4 launched in March 2023 with 8,192 tokens, which are small units of text. GPT 5.6 Sol handles around 1 million tokens, about 120 times as much.

What is not known

The report gives no names of tests, no data on the models and no numbers for cost. The team tells “in many tests”, not always.

Insight: Vendors sell more context while Nvidia questions the need

In October 2026, OpenAI increased a default context from 272k to 500k. In November 2026, Google added training on inputs of 14 million tokens. Those moves push for more context, but the team above tells that it is frequently overkill. Cost for each task possibly becomes the test.

Sources

  1. Nvidia researchers argue huge AI context windows are a dead endthe-decoder.com
AI MATTER · NEWS · AI MATTER · NEWS ·21 NOV2026

Posted

Tags

Learn the terms in this story

More in Research

All Research news