Chapter 3: Multimodal AI — Vision, Audio, Code, and Documents — The PM Version

Introduction

In Chapter 2, we compared model families — GPT, Claude, Gemini, Llama — and how to pick a provider for a use case.

Most real products are not text-only. Users upload PDFs, photos of forms, call recordings, screenshots, spreadsheets, and code repos. “Multimodal” is the label vendors use when a model can accept more than plain text in a single request.

That sounds simple. In production it is not. The same model that summarizes a clean PDF may misread a stamped invoice, skip a chart legend, or confidently describe an image detail that is not there.

This chapter is the PM map: what each modality is good for, where it breaks, how workflows should look, and when a dedicated OCR or speech pipeline beats “just send it to the vision model.”

1. What Multimodal Means

A multimodal model can take inputs beyond text — commonly images, audio, and sometimes video — and produce text (or other outputs) in one pass. Under the hood, non-text inputs are usually converted into token-like representations the transformer can attend over, alongside your prompt.

“Multimodal” is not one capability level. Vendors differ on:

• which file types are supported,
• max resolution and page count,
• whether audio is native or transcribed first,
• whether code is “seen” as files or only pasted snippets,
• pricing per image, per minute, or per token.

For PMs, the useful definition is operational: Can my product send this asset type to the model API in one call, and get a structured answer back? If yes, you still must ask whether that path is accurate and affordable at your volume.

2. Why PMs Should Care

Multimodal unlocks features that text-only chat cannot ship: “upload your bill and we explain the line items,” “summarize this call,” “review this UI screenshot,” “explain this repo.” It also expands failure surface — privacy, latency, cost, and wrong extractions that look authoritative.

3. Modality Comparison — Strengths and Tradeoffs

Use this table in roadmap discussions — not as gospel, but to force explicit tradeoffs per feature.

4. Vision Use Cases — Images and Screenshots

Vision is strongest when the task is semantic understanding of what is visible, not when you need pixel-perfect transcription of a 40-column spreadsheet.

5. Document Use Cases — PDFs, Scans, and Forms

Documents are where multimodal hype meets operations reality. Claims, invoices, contracts, and medical records mix text, tables, stamps, and handwriting.

Connects to grounding lessons from Module 01 — Why LLMs Hallucinate: extraction without citation is still hallucination risk, just with a PDF attached.

6. Audio Use Cases — Voice and Recordings

Audio products usually chain speech-to-text then LLM reasoning, unless the vendor offers a single native audio endpoint. PMs should spec both steps — transcript quality caps everything downstream.

7. Code Use Cases — Repos, Diffs, and Logs

Code is text structurally, but product-wise it behaves like a modality: large files, strict syntax, and verifiable truth via compilers and tests.

8. Limitations — What Breaks in Production

Multimodal does not fix core LLM limits. It adds perception errors on top of reasoning errors.

9. PM Workflows — From Demo to Production

A credible multimodal feature usually follows this workflow — adjust names to your org, keep the gates.

Example: claims document intake

1. User uploads bill + discharge summary.
2. System classifies doc types and runs extraction (pipeline or multimodal).
3. Validator checks required fields and formats.
4. Model drafts summary with citations to page/line.
5. Processor reviews low-confidence fields in side-by-side UI.
6. Approved fields write to claim system of record.

10. Design Checklist — Before You Ship

Use this checklist in design review. Missing items are how multimodal pilots die in production.

11. Multimodal vs OCR — When to Use Which

OCR (optical character recognition) and layout parsers extract text and structure first; the LLM reasons on text. Native multimodal sends pixels or pages to the model directly.

Common winning pattern: OCR or layout extraction for fields + multimodal or LLM for summary and classification — with one golden eval set covering the full chain.

Chapter Summary

Closing Thought

Multimodal is why many AI products finally feel “real” to business users — the model meets them where their work already lives, in files and recordings.

It is also where demos diverge fastest from production. A slick upload button does not replace field validation, source highlighting, or reviewer queues.

As a PM, your job is to name the modality, name the acceptable error rate, and design the human and system checks around perception — before you argue about which logo is on the model card.

Next we look at benchmarks — what public scores actually measure, what they miss, and how to build evaluation your stakeholders should trust more than a leaderboard screenshot.

Chapter navigation