A Google team makes a video agent that selects when to speak

A team from Google, Stanford, Oxford and other US universities made StreamVibe. It is a video agent, which is software that monitors a user at work and selects when to speak. The user can give a general instruction about help with a problem. The authors write that the model improves the time and quality of replies by 10.7 percent.

How it works

The team made StreamVibe-VL. It is possible that it uses the Alibaba model Qwen3.5-9B. Each second, it selects if it waits or speaks. A next step of training makes it speak less. A different memory records which instructions are open and which steps are done.

The numbers

The test set has about 4,100 examples from more than 183 hours of video. The authors write that the memory adds 12.3 percent to help that starts without an instruction. Gemini 3.8 Flash, a Google model that is not open, scores more in some tests.

What is not known

In a full video, all models in the test make more errors. One score of the replies decreases by a maximum of 61.5 percent. After five minutes, the models do not frequently find the correct time to speak. The authors write that usual use can be not as easy as their test.

Insight: Google puts checks around the agent

In other Google work, a lab simulator rejects commands that cannot work. An app stops screen parts that are not approved. Here, the models make more errors when they use their replies. Google puts checks around the agent, and this reads as a way to set the standard. Find who controls the checks in your agent tools.