Oikantik Nath, Hanani Bathina, Mohammed Safi Ur Rahman Khan, Mitesh M. Khapra
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent advancements in Vision-Language Models (VLMs) have opened new possibilities in automatic grading of handwritten student responses, particularly in mathematics. However, a comprehensive study to test the ability of VLMs to evaluate and reason over handwritten content remains absent. To address this gap, we introduce FERMAT, a benchmark designed to assess VLMs' ability to detect, localize and correct errors in handwritten mathematical content. FERMAT spans four key error dimensions -computational, conceptual, notational, and presentation -and comprises over 2,200 handwritten math solutions derived from 609 manually curated problems from grades 7-12 with intentionally introduced perturbations. Using FER-MAT we benchmark nine VLMs across three tasks: error detection, localization, and correction. Our results reveal significant shortcomings in current VLMs in reasoning over handwritten text, with GEMINI-1.5-PRO achieving the highest error correction rate (77%). We also observed that some models struggle with processing handwritten content, as their accuracy improves when handwritten inputs are replaced with printed text or images. These findings highlight the limitations of current VLMs and reveal new avenues for improvement. We release FERMAT and all the associated resources in the open-source to drive further research. Perturbation Axes # Inst Perturbation Description COMPUTATIONAL (CO) 611 CALCULATION & PROPAGATION ERRORS FINAL NUMBER 156 Incorrect final answer including digit swaps or misplaced decimals. INTERMEDIATE CALCULATION 100 Arithmetic calculation errors in intermediate steps. NON-PROPAGATED STEP ERROR 80 Error in intermediate step corrected in subsequent steps. PROPAGATED STEP ERROR 108 Error in intermediate step carried forward. COPY ERROR 167 Copying wrong numbers/expressions from question (e.g., copying 45 as 54). CONCEPTUAL (CP) 609 INCORRECT INTERPRETATION OF CONCEPTS THEOREM MISAPPLICATION 62 Applying theorems/identities incorrectly (e.g., using sin 2 θ + cos 2 θ = 0). MISINTERPRET QUESTION 145 Misreading problem requirements such as reporting area instead of volume. INVALID ASSUMPTION 122 Making assumptions without justification/verification. OUTRIGHT INCORRECT FACT 143 Stating objectively false information (e.g., a triangle has two right angles). FORMULA MISUSE 137 Incorrectly writing a standard formula (e.g., Circle Area: πr 2 → 2πr). NOTATIONAL (NO) 255 MISTAKES IN MATH SYMBOLS & OPERATORS SYMBOL ERROR 81 Mistakes in symbols/notation (e.g., x 2 → x2). OPERATOR SWAP 115 Incorrect substitution of operators (e.g., + → ×). MISPLACED PARENTHESES 59 Misplacing parentheses, thus changing the intended order of operations. PRESENTATION (PR) 429 ISSUES IN FORMATTING & LOGICAL FLOW FORMAT IGNORED 47 Ignoring question-specified format (e.g., standard vs scientific notation). TERMINOLOGY SWAP 25 Switching inconsistently between terms (e.g., "vector" ←→ "line"). LOGIC DISRUPTION 101 Presenting steps out of logical order (e.g., final answer used in earlier steps). CONTEXTUAL SWAP 43 Contextually similar but incorrect term substitution (e.g., circle → ellipse). VARIABLE MISNAMING 67 Swapped variables (e.g., swapping a and b in a quadratic formula). INCORRECT UNITS 146 Reporting with wrong units (e.g., length in kg instead of m).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2501.07244")
get_code_for_paper("2501.07244")
have("2501.07244")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2501.07244.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2501.07244)
Connect an agent — have() is free.