Multimodal model

How a multimodal model works

Models connect types of data in different ways. LLaVA, for example, changes image features into language embedding tokens. Three models show three methods:

Model	Year	Method
CLIP	2021	Learns which caption goes with which image, from 400 million image and text pairs
Flamingo	2022	Connects a pretrained image model to a language model. Reads images and text mixed
LLaVA	2023	Sends CLIP image features through one trainable matrix into the Vicuna language model as tokens

Some models learn from all types of data from the start. OpenAI says GPT-4o is “trained end-to-end across text, vision, and audio”. It answers audio in 232 milliseconds at the fastest, 320 on average (system card, October 2024).

Inputs and outputs of known models

The word “multimodal” does not tell which types go in or come out:

• GPT-4 (OpenAI, March 2023): image and text in, text out.
• Gemini (Google, December 2023): image, audio, video and text in. Three sizes: Ultra, Pro and Nano.
• GPT-4o (OpenAI system card, October 2024): text, audio, image and video in. Text, audio and image out.

Why multimodal models are a security risk

Each new type of input is a new path for an attacker. A text filter cannot see an instruction in the pixels of an image.

1. Hidden instructions. Bagdasaryan and colleagues (July 2023) put instructions into images and sounds. LLaVA and PandaGPT then gave text that the attacker chose. This is indirect prompt injection through a picture.
2. Image jailbreaks. Qi and colleagues (June 2023) found that “a single visual adversarial example can universally jailbreak an aligned LLM”. This is an adversarial example used for jailbreaking.
3. No mature defence. OWASP LLM01:2025 says that cross-modal attacks are “difficult to detect and mitigate with current techniques”.

Questions and answers

What is the difference between a multimodal model and a language model?

A language model reads text and gives text. A multimodal model also reads other types of data, such as images, audio or video. Many multimodal models are a language model with an added encoder for images or sound. LLaVA, for example, connects the CLIP image encoder to the Vicuna language model.

Can an image attack a multimodal model?

Yes. In 2023, Bagdasaryan and colleagues hid instructions in images and audio. These steered LLaVA and PandaGPT to give text that the attacker chose. Qi and colleagues showed that one adversarial image can jailbreak an aligned model. OWASP puts hidden instructions in images in prompt injection.