AI Models Struggle with Medical Complexity
Medical professionals and insurance claims adjusters often face daunting tasks when dealing with lengthy and complex medical case files. A recent leaderboard from Wisedocs' Medical Long Context Reasoning benchmark, launched on August 21 by Artificial Analysis, sheds light on the performance of AI models in this area.
The top-performing model, Anthropic's Claude Fable 5, achieved a score of 64.4%, which might not seem impressive at first but is significantly higher than the median model's score of below 15%.
The benchmark, designed by Wisedocs from its deep domain knowledge gained through training on over 100 million claim documents, includes 250 questions spread across six difficulty tiers. The leaderboard specifically tests models on the two hardest tiers, Expert and Compound, using 60 synthetic medical and insurance case questions that average 70 to 150 pages in length.
One of the interesting findings from the leaderboard is a tension between accuracy and completeness. While many models excel at getting individual answers right, they struggle with answering all the questions a case file demands. For instance, GPT-5.6 Terra (max) posted the highest accuracy score at 93.7% but ranked only 10th overall due to its low completeness.