Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

摘要

Multimodal Large Language Models (MLLMs) have beengrowing the capability for scienti_x001C_c writing and collaboration. For exam-ple, OpenAI Prism is a free workspace for scienti_x001C_c writing and collab-oration. One important feature in Prism is turning scienti_x001C_c diagramsdirectly into LATEX TikZ code. In this paper, we build a benchmark,Diagram-MMU, a multi-modal benchmark designed to assess MLLMs'ability for scienti_x001C_c diagram parsing and understanding. Diagram-MMUfeatures 3.7k curated diagrams and 18.3k human-validated questionsacross six domains. It evaluates MLLMs on three tasks common in vibewriting workspaces: diagram-to-code parsing, diagram-to-code editing,and diagram question answering, alongside agentic settings per task.The evaluation of 12 MLLMs reveals that diagram-to-code tasks aremore challenging than diagram question answering: models can reasonwell over diagrams but struggle to parse and edit them, underscoringthe need for methods to enhance MLLMs' capability in diagram-to-codegeneration. Under agentic settings, most models improve parsing andediting performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Our benchmark ispublicly available at https://huggingface.co/datasets/AIGrounding/Diagram-MMU.