A small text model aligns more with image AI than a 72B model
The paper says that training on two views of one scene uses 42 times less data than next token prediction for the same alignment.
A new paper turns the Platonic Representation Hypothesis into a theorem. The hypothesis says that models learn the same picture of the world from different types of data, when they get more parameters. The paper proves it for one type of model, JEPA. It says that its text model with 109 million parameters aligns with image encoders more than language models as large as 72 billion.
How it works
A JEPA learns to predict one view of a scene from a different view. The paper proves that JEPAs learn the same representation, with one rotation as the only difference, when the pairs are good views. Next token prediction means that a model predicts the next piece of text. The paper says that it is a JEPA with bad views, because it uses parts of one document.
The numbers
- Good views use 42 times less training data than next token prediction.
- The text model with 109 million parameters aligns more than each language model.
- Without pairs, it gets 64% on ImageNet, against at most 56% for language models as large as 8 billion.
What is not known
All results come from the authors. The text gives no tests by other teams. For other pairs, the misalignment has a limit.
Insight: A good training goal can give more than more parameters
The paper says that good views, not only scale, make the same representation in models. Other research connects hard to read AI text to training for math and code, and says that training with a clear goal can possibly repair it. Maybe the two show that a good goal of training can give more than a large number of parameters. Teams that select the training goal can be first.
Sources
Posted
