A 2015 network beat two top AI models on skin lesions, 62% to 42%

An author tested Inception v3, a convolutional neural network (an image model) that Google published in 2015, against GPT-6 Astra and Claude Fable 5.1. All three saw 200 skin lesions with a biopsy result. The author tuned the small network for about twenty minutes, and it named the right diagnosis for 62%, against 42% for the two language models.

How the test worked

Each lesion had two images: a close-up photograph and a dermoscopic image, made with a skin magnifier. Each model chose from six diagnoses. The language models got the same prompt, and the author removed the test lesions from the training data of the network.

The numbers

The network named 13 of the 30 melanomas, and the language models named 8 and 11. Each language model had a favorite incorrect answer. GPT-6 Astra called 75 lesions basal cell carcinoma, and 25 were.

What is not known

• 30 melanomas give wide uncertainty, and answers can change between runs.
• All lesions had a biopsy, which makes them hard cases.
• The test data is public, and the models can know it.
• The author says regulated tools built on better methods do much better.

Insight: A test on your own data beats a brand name

Mayo Clinic made a model for pancreatic cancer risk but gave no result for how correct it is. Here the author gives the test lesions, the prompts, all replies and the scoring code. It reads as a move of control from the biggest model to the one who owns task data and an open test. Buyers of AI for a narrow task must tell vendors to show such a test.