A 10-year-old CNN beat frontier LLMs at reading skin cancer

Magnus Lynch, a dermatologist, ran a small but careful test of whether frontier language models can read skin lesions from images, and compared them with a convolutional neural network from a decade ago. He used 200 biopsy-proven lesions from the public MILK10k dataset, half of them cases that should not be sent home (melanoma and other cancers) and half safe to discharge. GPT-6 Astra and Claude Fable 5.1 each got the diagnosis right as their first choice 42% of the time. The CNN, an Inception v3 model fine-tuned on about 68,000 dermatology images, scored 62%.

The failure modes are the interesting part. The two LLMs were cautious to a fault: they kept almost every dangerous lesion (97 and 96 of 100) but also flagged a third of the harmless ones, and in a patient-style test on clinical photos they told nearly everyone to see a doctor, including people with obviously benign spots. The CNN was more decisive and more accurate, but at its preset threshold it discharged 18 of 30 melanomas, which is its own kind of danger. Lynch is clear about the limits: the sample is small, each model was run once, the models may have seen the public dataset in training, and Inception v3 is a baseline rather than a current specialist system.

Why it matters

If you are deciding whether to swap a working task-specific model for a general LLM, this is a useful check: on a narrow perception task a cheap, old, specialized network still beat two frontier models by 20 points, so test on your own labeled data before you assume the newer, more expensive model is better.