Photo2Math

1. Core Problem & Scope

Converting unstructured handwritten mathematical expressions into interactive graphs requires handling varying spatial stroke arrangements — fractions, exponents, and nested expressions don't follow the simple left-to-right layout that standard OCR assumes. The scope was to take a single photo of a handwritten expression (or line drawing) and produce a live, explorable graph rendering of the underlying equation.

2. Comparative System Research

• Traditional OCR + regex parsing — works for simple, cleanly-written linear expressions, but breaks down entirely on two-dimensional layouts like fractions and nested exponents, which standard OCR has no concept of.
• Custom CNN symbol classifier + hand-built layout rules — more tractable for 2D layout, but requires extensive rule engineering for every new expression pattern and doesn't generalize well to messy handwriting.
• Vision Transformer OCR (TrOCR) directly to LaTeX (chosen) — a transformer trained to output LaTeX token sequences directly learns the two-dimensional structural conventions of mathematical notation as part of its sequence generation, rather than requiring hand-built layout rules.

3. Architectural Pipeline Layouts

Handwritten Image Capture → Vision Transformer (TrOCR) → LaTeX Token Output → Abstract Syntax Tree (AST) → Dynamic MathJS Graph Render

A photographed expression is passed through a TrOCR-based vision transformer that outputs a LaTeX token sequence directly, capturing structural elements like fractions and exponents in the token stream itself. The LaTeX output is compiled into an Abstract Syntax Tree to resolve operator precedence and nested expression structure unambiguously, and the AST drives a live MathJS-rendered, explorable graph.

4. Trade-off Engineering Logs

Why LaTeX as the intermediate representation instead of a custom symbolic format: LaTeX already has a mature, well-specified grammar for exactly the two-dimensional constructs this problem needs to handle (fractions, exponents, roots, subscripts), and existing parsers can compile it into an AST — building a custom intermediate format would have meant re-solving a problem LaTeX's grammar already solves well.

AST compilation as a separate step from OCR, rather than rendering LaTeX directly: Rendering LaTeX directly would produce a static equation image, not an explorable graph. Compiling to an AST first is what allows the system to evaluate the expression symbolically and drive an interactive MathJS visualization instead of a fixed rendering.

5. Scaling & Production Failures

Early TrOCR outputs on messier handwriting occasionally produced syntactically invalid LaTeX — mismatched braces or malformed fraction structures — which broke the downstream AST compilation step. Adding a LaTeX validation and auto-correction pass between OCR output and AST compilation (catching and repairing common malformed patterns like unmatched braces) substantially reduced hard failures on real handwritten input, turning outright parse failures into either a successful render or a clear "couldn't parse" response instead of a silent crash.