Free lesson · AI Engineering
Vision Models and Image Understanding
Vision Models and Image Understanding
The Feature That Made Engineers Stop Taking Screenshots of Error Messages
At Meta, engineers had a peculiar habit that drove the developer tools team crazy. When they encountered an error in a monitoring dashboard, they would take a screenshot of the dashboard, paste it into a Slack thread, and type "what does this mean?" The error message was text on screen - perfectly copyable, searchable text. But engineers screenshotted it because taking a screenshot was faster than reading the dashboard, finding the error ID, looking up the error code in documentation, and interpreting the context around it.
We built a feature that accepted screenshot uploads and used GPT-4 Vision to analyse them. The user pasted a screenshot of a monitoring dashboard. The vision model identified the error message, read the surrounding metrics, understood the dashboard layout, and produced an interpretation: "Your service is returning 502 errors at a rate of 14% over the last 15 minutes. The error correlates with a memory spike starting at 10:23 AM. This pattern typically indicates an out-of-memory condition in the application container." The response included the error code, the relevant metrics, and a suggested investigation path - all extracted from a screenshot that contained text the user could have copied manually but chose not to.
This feature succeeded not because vision models are better than text search - they are not. It succeeded because it matched the user's natural behaviour (screenshotting) rather than requiring the user to change their behaviour (manually copying error codes). That insight - meeting users where they are rather than where your architecture wants them - is the foundation of every successful vision model application.
How Vision Models Work
Vision models process images alongside text, enabling applications that can see and reason about visual content. The technical architecture varies by model family, but the fundamental approach is shared: the image is encoded into a sequence of tokens (similar to how text is tokenised), and those visual tokens are processed by the transformer architecture alongside the text tokens.
The key models in 2025:
GPT-4V (OpenAI) and GPT-4o. OpenAI's vision-capable models accept images as part of the message input. GPT-4o processes images natively with the same architecture that handles text, enabling tight integration between visual and textual reasoning. Strong at detailed image description, OCR, chart reading, and visual question answering. Accepts multiple images per request.
Claude 3.5 Sonnet Vision (Anthropic). Anthropic's vision models excel at document understanding, technical diagram interpretation, and structured data extraction from images. Particularly strong at reading dense technical content - code screenshots, architecture diagrams, spreadsheet images - where precise character-level accuracy matters.
Gemini Pro Vision (Google). Google's multimodal model processes text, images, audio, and video in a unified architecture. Strong at real-world image understanding and particularly capable with video input (discussed in Topic 17.4). The native multimodality means visual and textual reasoning are deeply integrated rather than bolted together.
Open-source vision models. LLaVA (Large Language and Vision Assistant), InternVL, and Qwen-VL provide vision capabilities with self-hostable models. Quality is approaching but not yet matching proprietary models on complex visual reasoning tasks. The gap is closing with each generation - LLaVA-1.6 and InternVL-2 produce results within 10-15% of GPT-4V on standard benchmarks.
graph TD
subgraph Input["Image Input Processing"]
I1["Raw image\n(PNG, JPEG, WebP)"]
I2["Image encoder\n(ViT, CLIP, or native)"]
I3["Visual tokens\n(image → token sequence)"]
end
subgraph Combined["Combined Processing"]
C1["Visual tokens + text tokens\nprocessed together by\ntransformer architecture"]
C2["Cross-attention between\nvisual and textual features"]
end
subgraph Output["Output"]
O1["Text response grounded\nin visual understanding"]
end
subgraph Use_Cases["Production Use Cases"]
U1["Document OCR\nand extraction"]
U2["Dashboard/chart\ninterpretation"]
U3["Visual QA\nand description"]
U4["Technical diagram\nanalysis"]
end
I1 --> I2 --> I3
I3 --> C1
C1 --> C2
C2 --> O1
O1 --> Use_Cases
style Input fill:#1e1b4b,stroke:#818cf8,stroke-width:2px
style Combined fill:#312e81,stroke:#a5b4fc,stroke-width:2px
style Use_Cases fill:#166534,stroke:#86efac,stroke-width:2px
This diagram shows how vision models process images. A raw image in PNG, JPEG, or WebP format passes through an image encoder (ViT, CLIP, or native) which converts it into a sequence of visual tokens. These visual tokens are combined with text tokens and processed together by the transformer architecture, with cross-attention between visual and textual features. The output is a text response grounded in visual understanding. Production use cases include document OCR and extraction, dashboard and chart interpretation, visual question answering and description, and technical diagram analysis.
Practical Use Cases That Work in Production
Vision models are not universally reliable. Some use cases produce consistently good results. Others produce results that are impressive in demos but fail under production conditions. Knowing the difference is critical before investing engineering effort.
Use Case 1: Document OCR and Structured Extraction
What it does: Extract text and structured data from images of documents - invoices, receipts, forms, ID cards, business cards, handwritten notes. The vision model reads the document, identifies fields, and outputs structured data (JSON with field names and values).
Why it works well: Documents have predictable layouts with labelled fields. The text is usually clear and high-contrast. The extraction task is well-defined - extract specific fields and their values. Evaluation is straightforward - the extracted fields either match the ground truth or they do not.
Production accuracy: 92-97% field-level accuracy for printed documents with standard layouts. 80-90% for handwritten text. 70-85% for complex layouts with nested tables, multi-column formats, or overlapping elements. Accuracy drops significantly for low-resolution images (below 300 DPI) and for documents with background patterns or watermarks.
Implementation pattern: Send the document image with a structured extraction prompt that specifies the expected fields. Use JSON output format for machine-readable results.
async def extract_invoice_data(image_bytes: bytes) -> dict:
"""Extract structured data from an invoice image."""
response = await client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{base64.b64encode(image_bytes).decode()}",
"detail": "high" # High resolution for document OCR
}
},
{
"type": "text",
"text": """Extract the following fields from this invoice image.
Return ONLY valid JSON with these fields:
{
"vendor_name": string,
"invoice_number": string,
"invoice_date": string (YYYY-MM-DD),
"total_amount": number,
"currency": string (3-letter code),
"line_items": [{"description": string, "quantity": number, "unit_price": number}]
}
If a field is not visible or unreadable, use null."""
}
]
}],
response_format={"type": "json_object"},
max_tokens=1000
)
return json.loads(response.choices[0].message.content)
Use Case 2: Chart and Dashboard Interpretation
What it does: Read charts, graphs, and monitoring dashboards from screenshots or images. Extract data points, identify trends, and summarise findings in natural language.
Why it works well: Charts encode data visually in standardised formats (bar charts, line graphs, pie charts). Vision models recognise these formats and can extract approximate values. The task maps well to how humans read charts - identify axes, read data points, describe trends.
Production accuracy: Vision models reliably identify chart types, read axis labels, and describe overall trends (up, down, stable). They are less reliable at reading precise numerical values from charts - a bar chart value of "14.7%" might be read as "approximately 15%." For applications where approximate values are sufficient (trend analysis, anomaly detection), accuracy is excellent. For applications requiring exact values (financial reporting, regulatory compliance), vision model output should be treated as approximate and verified against the data source.
Use Case 3: Visual Question Answering
What it does: Answer natural language questions about image content. "What brand is this product?" "How many people are in this photo?" "Is this a cat or a dog?" "What material is this surface made of?"
Why it works well for simple questions: Questions with clear, unambiguous answers about visible content produce reliable results. Object identification, colour recognition, text reading, counting (for small numbers), and spatial relationship questions ("is the red box above or below the blue box?") are handled well.
Where it fails: Questions requiring domain expertise ("What geological formation is this?"), precise counting of large numbers ("How many people are in this stadium?"), fine-grained visual distinctions ("Is this a 10mm or 12mm wrench?"), and subjective judgments ("Does this person look happy?") are unreliable. The model's confidence does not correlate with accuracy on these harder questions - it answers confidently even when wrong.
Use Case 4: Technical Diagram Analysis
What it does: Interpret architecture diagrams, flowcharts, UML diagrams, network topologies, and other technical visualisations. Identify components, connections, data flows, and relationships.
Why it works for specific diagram types: Flowcharts and simple architecture diagrams with clear boxes and arrows produce good results - the model identifies components and their connections accurately. Sequence diagrams and state diagrams are handled reasonably well. Complex diagrams with many overlapping elements, small text, or non-standard notation are less reliable.
At Meta, we built a feature that analysed architecture diagrams submitted for design review. Engineers uploaded a diagram, and the vision model described the architecture: "This diagram shows a three-tier architecture with a React frontend communicating via REST API with a Node.js backend. The backend connects to a PostgreSQL primary database with a Redis cache layer. There is a message queue (shown as Kafka) connecting the backend to an async processing service." The descriptions were accurate 85% of the time for standard architecture diagrams with clear labelling. For hand-drawn whiteboard diagrams, accuracy dropped to 60%.
Limitations Every AI Engineer Must Know
Limitation 1: Hallucinated Text
Vision models sometimes "read" text that is not in the image. A blurry sign might be interpreted as specific text that looks plausible but is wrong. A partially obscured label might be completed with fabricated characters. This is the visual equivalent of text hallucination - the model generates the most probable text rather than admitting it cannot read the content clearly.
Mitigation: For applications where text accuracy is critical (OCR, document extraction), implement confidence thresholds. When the model's confidence is below the threshold, flag the extraction as "uncertain" and route to human review. Never trust vision model OCR output for legally binding documents without human verification.
Limitation 2: Spatial Reasoning Failures
Vision models struggle with precise spatial relationships, measurements, and counting. "How far apart are these two objects?" "What is the exact angle of this line?" "How many windows are on this building?" These questions require spatial reasoning that current vision models perform unreliably.
Mitigation: Do not use vision models for tasks requiring precise spatial measurements. Use specialised computer vision models (object detection, segmentation) for counting and spatial tasks, and vision language models for interpretation and reasoning.
Limitation 3: Resolution and Detail Sensitivity
Vision models process images at a fixed internal resolution. Sending a 4K screenshot of a monitoring dashboard means the model sees a compressed representation - fine details like small text labels, thin lines, and subtle colour differences may be lost or misread. The detail parameter (available in OpenAI's API: "low" = 512px, "high" = 2048px max) controls the processing resolution and directly affects both accuracy and cost.
Mitigation: Crop images to the relevant region before sending. A full-screen dashboard screenshot contains mostly irrelevant whitespace - cropping to the chart of interest reduces visual noise and improves accuracy. For documents, send individual pages rather than full multi-page spreads.
Limitation 4: Cost
Vision model calls are significantly more expensive than text-only calls. A single image adds 85-1,600 tokens (depending on resolution and the model's image tokenisation) to the request. At high resolution, a single image can cost $0.01-0.04 to process. For applications processing thousands of images per day, vision model costs can dominate the inference budget.
Mitigation: Use the lowest resolution setting that produces acceptable accuracy for your task. Process images with traditional computer vision first (edge detection, region of interest extraction) and send only the relevant portions to the vision model. Cache vision model outputs for identical or near-identical images.
graph TD
subgraph Works["✓ Works Well in Production"]
W1["Document OCR\n92-97% on printed docs"]
W2["Chart trend analysis\nReliable for directions, approx values"]
W3["Simple visual QA\nObject ID, colour, text reading"]
W4["Standard architecture diagrams\n85% accuracy with clear labels"]
end
subgraph Unreliable["△ Unreliable - Use With Caution"]
U1["Precise number reading\nfrom charts"]
U2["Handwritten text\n80-90%, varies heavily"]
U3["Complex/dense diagrams\n60% on whiteboard drawings"]
U4["Fine-grained distinctions\n(10mm vs 12mm wrench)"]
end
subgraph Fails["✗ Do Not Use For"]
F1["Precise spatial measurement"]
F2["Large-number counting"]
F3["Legally binding OCR\n(without human review)"]
F4["Domain-expert visual analysis\n(medical imaging, geology)"]
end
style Works fill:#166534,stroke:#86efac,stroke-width:2px
style Unreliable fill:#854d0e,stroke:#fde047,stroke-width:2px
style Fails fill:#991b1b,stroke:#fca5a5,stroke-width:2px
This diagram categorises vision model capabilities into three reliability tiers. Works well in production: document OCR at 92 to 97 percent on printed documents, chart trend analysis for directions and approximate values, simple visual QA for object identification and text reading, and standard architecture diagrams at 85 percent accuracy with clear labels. Unreliable and to be used with caution: precise number reading from charts, handwritten text at 80 to 90 percent with heavy variation, complex dense diagrams at 60 percent for whiteboard drawings, and fine-grained visual distinctions. Do not use for: precise spatial measurement, large-number counting, legally binding OCR without human review, and domain-expert visual analysis like medical imaging.
Interview: System design interview connection: "How would you add image understanding to this application?" is an increasingly common question as vision models become standard. The interviewer wants to see that you understand the capabilities and limitations - not just that vision models exist.
What to emphasise: Name specific models and their strengths (GPT-4o for general vision, Claude for document understanding, Gemini for video). Present the four production use cases with accuracy ranges. Discuss the limitations (hallucinated text, spatial reasoning, resolution sensitivity, cost) and their mitigations. The strongest signal is when you recommend against using vision models for specific sub-tasks where traditional CV is more appropriate.
What separates strong from weak answers: Weak answers describe sending an image to GPT-4V and getting a response. Strong answers specify the resolution setting, the prompt structure for extraction, the accuracy range for the task type, the cost per image, and the limitations that affect the specific use case. The strongest answers propose a hybrid architecture: traditional CV for detection/segmentation/counting, vision LLM for interpretation/reasoning.
Exercise: Starter: Your company receives 5,000 invoice images per day via email. Design the automated extraction pipeline using a vision model. What fields do you extract? What accuracy do you need? What resolution setting do you use? What happens when the model cannot read a field?
Intermediate: Your monitoring platform wants to add a "screenshot analysis" feature where users upload dashboard screenshots and get AI interpretation. Design the feature: what vision model, what prompt structure, what limitations to communicate to users, and what accuracy metrics to track. How do you handle the case where the screenshot contains PII (customer names, IP addresses)?
Advanced: Design a multimodal document processing pipeline that handles PDFs (text extraction), scanned documents (OCR), photographs of whiteboards (handwriting recognition + diagram interpretation), and screenshots of code (code extraction + language detection). Each input type requires different processing. Include: the input classifier (how you determine input type), the processing pipeline per type, the accuracy targets per type, the cost model, and the evaluation strategy.
Key Takeaways
Vision models enable applications that process images alongside text - document OCR, chart interpretation, visual Q&A, and technical diagram analysis. The key models in 2025 are GPT-4o (strong general vision), Claude 3.5 Sonnet Vision (strong document understanding), Gemini Pro Vision (native multimodality with video support), and open-source options like LLaVA and InternVL (self-hostable, approaching proprietary quality).
Production accuracy varies dramatically by use case. Document OCR on printed text achieves 92-97% field accuracy. Chart trend analysis is reliable for directions and approximate values but unreliable for precise numbers. Technical diagram analysis is 85% accurate for standard diagrams with clear labels but 60% for complex or hand-drawn diagrams. Knowing these accuracy ranges before building is what prevents the gap between demo quality and production quality.
Four limitations constrain every vision model application: hallucinated text (the model reads text that is not there), spatial reasoning failures (imprecise measurements and counting), resolution sensitivity (fine details lost at lower resolutions), and cost (vision calls are 5-40x more expensive than text-only calls). Each limitation has specific mitigations - confidence thresholds, traditional CV for spatial tasks, image cropping, and resolution optimisation.
The most successful vision model applications meet users where they are - accepting screenshots, photos, and visual inputs that match natural user behaviour - rather than requiring users to convert visual information into text that the traditional system can process.