AI Interview Questions

A multimodal model is an AI system that can process and integrate information from multiple types of data (modalities) simultaneously, such as text, images, audio, and video. Unlike single-modality models (text-only LLMs), multimodal models can understand relationships across different data types.

Modern multimodal models handle various modalities: 1) Text - natural language understanding and generation. 2) Images - recognizing objects, scenes, text within images (OCR), visual relationships. 3) Audio - speech recognition, sound classification, music understanding. 4) Video - temporal understanding, action recognition, combining visual and audio streams. 5) Documents - understanding layouts, tables, charts, and text together. Architectures typically use separate encoders for each modality (e.g., vision transformer for images, text encoder for language) that project inputs into a shared representation space, allowing cross-modal reasoning. Models like GPT-4V, Gemini, and Claude 3 handle text+images; some also support audio and video.

Real-world data is multimodal - documents contain images, videos have audio, websites mix text and graphics. Multimodal models can understand this richness, enabling applications impossible with text-only: analyzing charts, describing images, transcribing and understanding videos, and answering questions about visual content. They bring AI closer to human-like perception.

User shows photo of a plant with yellowing leaves and asks 'What's wrong with my plant?' Multimodal model: sees image (yellow leaves, drooping), reads text. Combines to diagnose: 'This appears to be overwatering - the yellowing and drooping are classic symptoms. Reduce watering and ensure good drainage.' Text-only model couldn't see the leaves; image-only model couldn't provide diagnosis. Multimodal understanding enables this.

A vision-language model (VLM) is a type of multimodal model that understands and generates language based on visual input. It can describe images, answer questions about visual content, and perform tasks that require integrating vision and language.

VLMs typically consist of: 1) Vision encoder - usually a vision transformer (ViT) or CNN that converts images into a sequence of visual embeddings (patches). 2) Text encoder/decoder - a language model component that processes text. 3) Cross-modal fusion - mechanisms (cross-attention, joint embeddings) that align visual and textual representations. The model is trained on large datasets of image-text pairs (e.g., LAION-5B) using objectives like contrastive learning (CLIP) or generative next-token prediction (Flamingo, LLaVA). At inference, the model can accept both image and text inputs and generate text outputs.

VLMs enable AI to 'see' and communicate about what it sees. This unlocks countless applications: accessibility (describing images for blind users), content moderation (identifying harmful images), visual search (finding products by description), medical imaging (analyzing X-rays with reports), and robotics (understanding visual scenes). They bridge the gap between computer vision and natural language.

User uploads X-ray image and asks 'Are there any fractures?' VLM processes image through vision encoder, identifies bone structures, correlates with text query. Responds: 'Yes, there's a hairline fracture in the distal radius (wrist). The fracture appears non-displaced. Please consult a radiologist for confirmation.' This combines visual understanding with medical knowledge, providing valuable assistance.

While exact architectures are proprietary, modern multimodal models like GPT-4V, Gemini, and Claude 3's vision capabilities share common design patterns: they combine vision encoders with LLMs through cross-attention mechanisms, enabling the model to process images alongside text.

Typical architecture: 1) Vision encoder - a pre-trained vision transformer (ViT) or similar that splits images into patches and encodes them into visual tokens. 2) Projection layer - maps visual tokens into the LLM's embedding space. 3) LLM backbone - a transformer language model that can attend to both text tokens and visual tokens (via cross-attention or by concatenating visual tokens to the text sequence). 4) Training - multi-stage: initial contrastive pre-training (like CLIP), then supervised fine-tuning on vision-language tasks. Gemini is rumored to be trained natively multimodal from the start, rather than adding vision to a text model. This allows deeper integration.

Architectural choices affect capabilities. Models that add vision to existing LLMs (like GPT-4V) leverage strong language understanding but may have limitations in visual reasoning. Natively multimodal training (Gemini) potentially enables deeper cross-modal understanding. Understanding these differences helps select models for specific tasks.

When shown a graph and asked 'What was the revenue in Q3?', the model must: 1) Visually locate the Q3 bar. 2) Read the scale. 3) Extract the value. 4) Generate answer. This requires tight integration of vision and language. Architecture must support fine-grained visual understanding. Different models may excel at different aspects.

LLaVA (Large Language and Vision Assistant) is an open-source vision-language model that combines a vision encoder (CLIP) with a language model (Vicuna/LLaMA) using a simple projection layer. It demonstrates that strong vision-language capabilities can be achieved with relatively straightforward architecture and training.

LLaVA architecture: 1) Vision encoder - CLIP ViT-L/14 (pre-trained on image-text pairs). 2) Projection layer - a simple linear layer or small MLP that maps visual features into the LLM's embedding space. 3) Language model - Vicuna (fine-tuned LLaMA). Training process: Stage 1: pre-training for feature alignment - train only projection layer on image-caption pairs to align visual and text embeddings. Stage 2: end-to-end fine-tuning on instruction-following data, including visual conversations created by prompting GPT-4 with images and captions to generate diverse QA pairs. This generates 150k visual instruction examples.

LLaVA democratized vision-language AI. It showed that with clever data generation (using GPT-4 to create training data) and simple architecture, you can build capable VLMs at low cost. It achieved competitive performance with much larger proprietary models. LLaVA spawned many variants and became the foundation for open-source VLM research.

LLaVA can have a conversation about an image: User: 'What's in this image?' LLaVA: 'A person sitting on a bench in a park.' User: 'What season might it be?' LLaVA: 'The leaves on the trees are orange and red, suggesting autumn.' This conversational visual understanding came from training on GPT-4 generated conversations, demonstrating the power of synthetic data.

A visual encoder is a neural network that converts images into vector representations (embeddings) that capture visual features. CLIP (Contrastive Language-Image Pre-training) is a foundational model that trained visual and text encoders together to create aligned embeddings, and it's widely used as the vision backbone in multimodal LLMs.

CLIP trains two encoders: a vision encoder (ViT or ResNet) and a text encoder (Transformer) on 400M image-text pairs from the internet. The training objective: for each batch, maximize similarity (cosine) between correct image-text pairs and minimize similarity for incorrect pairs. This creates a shared embedding space where images and their descriptions are close. In multimodal LLMs, CLIP's vision encoder is often used as a frozen or fine-tuned component to provide visual features that the LLM can attend to. The pre-trained alignment means visual features already relate to language concepts.

CLIP revolutionized vision-language by providing high-quality, aligned visual representations without requiring expensive bounding box annotations. It's the backbone of most modern VLMs. Using CLIP saves enormous training compute and data. Its zero-shot capabilities (classifying images by text prompts) also enable flexible visual understanding.

In LLaVA, CLIP vision encoder processes an image of a dog, producing visual tokens. These tokens, already aligned with language (CLIP's text encoder would embed 'dog' close to this image), are projected and fed to the LLM. The LLM can then answer 'What animal is this?' because visual representation already encodes 'dog-ness'. CLIP provides this semantic visual understanding.

Vision-language models can perform a wide range of tasks that require understanding and generating language about visual content, from simple description to complex reasoning, visual question answering, and even creative tasks.

Task categories: 1) Visual question answering (VQA) - answer questions about images ('How many people are in this photo?'). 2) Image captioning - generate descriptions ('A dog playing in the park'). 3) Visual reasoning - more complex inferences ('Why is this person happy?'). 4) Document understanding - extract info from forms, charts, tables ('What was revenue in 2023?'). 5) Visual grounding - locate described objects ('Where is the red car?') - often outputs bounding boxes. 6) Multimodal dialogue - conversational QA about images. 7) Image-text retrieval - find images matching text or vice versa. 8) Visual storytelling - generate story from image sequence. 9) OCR and text recognition - read text from images. 10) Visual question generation - create questions about images.

These tasks span numerous applications: accessibility (describing images for blind), education (answering questions about diagrams), e-commerce (search by image), healthcare (analyzing medical images), content moderation (understanding image context), and robotics (scene understanding). VLMs bring AI's language understanding to the visual world.

User uploads screenshot of a graph showing company revenue over time and asks 'What was the growth rate between 2020 and 2023?' VLM: 1) Recognizes it's a line graph. 2) Locates 2020 and 2023 points. 3) Reads values (e.g., 100 and 150). 4) Calculates 50% growth. 5) Responds: 'Revenue grew 50% from 2020 to 2023.' This complex task combines visual recognition, text extraction, reasoning, and calculation.

Despite impressive capabilities, current VLMs have significant limitations: they can struggle with fine-grained visual details, counting, spatial reasoning, hallucinations, and domain-specific understanding. These limitations affect reliability in many applications.

Key limitations: 1) Fine-grained recognition - may confuse similar objects, breeds, models. 2) Counting - often inaccurate for more than a few objects. 3) Spatial reasoning - difficulty with relationships like 'left of', 'above'. 4) Hallucination - describing objects not present, especially when prompted suggestively. 5) Text recognition - OCR in complex layouts still error-prone. 6) Temporal understanding - struggle with video, change over time. 7) Compositionality - understanding complex relationships between multiple objects. 8) Domain shift - perform poorly on specialized domains (medical, satellite) without fine-tuning. 9) Bias - may reflect training data biases. 10) Computational cost - processing images is expensive.

Understanding limitations prevents over-reliance and guides appropriate use. For high-stakes applications (medical, autonomous driving), these limitations are critical. They also drive research: better architectures, more diverse training data, and specialized fine-tuning. Knowing what VLMs can't do is as important as knowing what they can.

VLM shown image of three dogs and asked 'How many dogs?' Might answer '2' or '4' - counting failure. Asked 'Is the dog on the left brown?' Might struggle with spatial reasoning. Asked about a rare medical condition in X-ray, may hallucinate. These limitations mean human oversight still needed. For counting tasks, consider combining with object detection models; for spatial reasoning, use specialized architectures.

Multimodal models handle video by extending image understanding across time, processing sequences of frames to capture motion, events, and temporal relationships. This is more complex than static images due to the temporal dimension and much larger data volume.

Approaches: 1) Frame sampling - select key frames at regular intervals (e.g., 1 frame per second) to reduce data. 2) 3D convolutions or video transformers - process spatio-temporal volumes directly. 3) Temporal attention - use attention mechanisms across frame embeddings to model relationships. 4) Audio integration - for videos with sound, incorporate audio modality. 5) Text generation - can summarize videos, answer questions about events, describe actions. Models like Gemini 1.5 Pro can process up to 1 hour of video by compressing visual information. Challenges: computational cost (many frames), temporal reasoning (understanding order, causality), and maintaining context across long videos.

Video is ubiquitous - surveillance, social media, education, entertainment. Video understanding enables applications: video search (find moments), summarization (create highlights), content moderation (detect harmful content), accessibility (describe videos for blind), and video QA (answer questions about content). As models improve, they unlock these capabilities.

User uploads 10-minute cooking video and asks 'What's the recipe and what were the key steps?' Video model: samples frames, recognizes ingredients, tracks actions (chopping, mixing), orders them temporally, extracts key moments. Responds with recipe summary and step-by-step instructions. This would be impossible with single-image models. Video understanding makes it possible.

Document understanding is the task of extracting and reasoning about information from documents that combine text, tables, images, and complex layouts (PDFs, forms, invoices, reports). Multimodal models approach this by jointly processing visual layout and textual content.

Approaches: 1) Layout-aware models (LayoutLM) - use bounding box coordinates along with text to understand spatial relationships. 2) Vision-language models - treat document page as image, use OCR to extract text, then reason about both visual and text elements. 3) Specialized document models (Donut) - end-to-end without OCR, reading text directly from image. 4) Hybrid - combine OCR for text extraction with visual understanding for layout, tables, figures. Models must handle: reading order, table structure, form fields, headers/footers, and mixed content. Key tasks: information extraction (invoices), document QA, document classification, table understanding.

Documents are the lifeblood of business - contracts, reports, invoices, forms. Automating document understanding saves enormous time and reduces errors. It enables search across scanned documents, automated data entry, and intelligent document processing. Multimodal models are revolutionizing this space.

Invoice processing: multimodal model receives scanned invoice PDF. It: 1) Identifies vendor name, invoice number, date (from layout). 2) Extracts line items from table (product, quantity, price). 3) Calculates subtotal, tax, total. 4) Outputs structured data. This task previously required template-based OCR or manual entry. Multimodal model adapts to varied invoice layouts, making automation robust.

Evaluating vision-language models requires benchmarks that test different capabilities: visual understanding, reasoning, grounding, and generation. Multiple datasets and metrics are needed because no single test captures all aspects.

Evaluation approaches: 1) VQA benchmarks (VQA v2, GQA) - test visual question answering accuracy. 2) Captioning metrics (CIDEr, SPICE, BLEU) - compare generated captions to references. 3) Visual reasoning (NLVR2, OK-VQA) - test reasoning requiring external knowledge. 4) Grounding (RefCOCO, Flickr30K) - test ability to locate described objects. 5) Document understanding (DocVQA, InfographicsVQA) - test complex document tasks. 6) Multimodal dialogue (MMDialog) - test conversational ability. 7) Human evaluation - rate helpfulness, accuracy, relevance. 8) Adversarial evaluation - test edge cases, robustness. Models are scored on each benchmark, often with leaderboards.

Different applications need different capabilities. A model good at VQA may fail at document understanding. Evaluation reveals strengths and weaknesses, guiding model selection and research. For practitioners, choosing the right benchmark for their use case is essential - don't use VQA to evaluate document understanding.

Choosing model for medical image analysis. Look at medical VQA benchmarks (VQA-Rad, PathVQA). Model A scores 85%, Model B 82%. But Model B better on radiology reports? Need to check. Also evaluate on out-of-distribution images, rare conditions. Comprehensive evaluation ensures chosen model meets actual needs, not just benchmark performance.

Early fusion combines different modalities at the input level, before any modality-specific processing. Late fusion processes each modality independently and combines their outputs at the decision level. Each has trade-offs in cross-modal interaction and computational efficiency.

Early fusion: raw data from different modalities (pixels, text) are combined early, often by concatenating or projecting into a joint representation before feeding into a unified model. Example: concatenating image patches with text tokens and processing together in a transformer. Enables deep cross-modal interactions but requires aligned data and can be computationally expensive. Late fusion: each modality processed by separate encoders, then their outputs (features, predictions) combined later via averaging, weighted sum, or another model. Simpler, allows independent pre-training, but misses fine-grained cross-modal interactions. Many modern VLMs use hybrid approaches: modality-specific encoders with cross-attention layers for interaction.

Choice affects model capability and complexity. Early fusion can capture subtle interactions (e.g., text referring to specific image regions) but harder to train. Late fusion easier but may miss cross-modal nuances. For tasks requiring fine-grained alignment (visual grounding), early fusion or cross-attention is essential. For simpler tasks, late fusion may suffice.

Visual question answering: 'What color is the car?' Late fusion: image encoder outputs features, text encoder outputs features, combined to predict answer. May miss that 'car' refers to specific region. Early fusion: image patches and text tokens together in transformer, allowing text to attend to relevant patches. Can ground 'car' to specific region and extract its color. Better for fine-grained tasks.

OCR (Optical Character Recognition) extracts text from images as raw characters, without understanding meaning or layout. Vision-language model document understanding goes further: it comprehends the document's structure, semantics, and can answer questions about content, not just extract text.

OCR: detects text regions, recognizes characters, outputs text string. May preserve rough layout but doesn't understand that a number is a price, or that text is a heading. Pure OCR is just text extraction. VLM document understanding: processes the document image holistically, understands that certain text is a header, numbers in a table relate to each other, can answer questions like 'What was the total revenue?' It combines visual layout understanding with semantic comprehension. Some VLMs use OCR as a preprocessing step, then reason about the extracted text with layout information.

For document automation, OCR alone is insufficient. Extracted text without structure is just a string - you can't reliably find the invoice total. VLM understanding enables true automation: extracting specific fields, answering questions, and performing complex document tasks. It's the difference between having the words and understanding the document.

Invoice document. OCR extracts: 'INVOICE #1234 Date: 2024-03-15 Item Qty Price Total Widgets 2 $10 $20 Gizmos 1 $15 $15 Subtotal $35 Tax $2.80 Total $37.80'. This is just text. VLM understanding: identifies 'INVOICE #1234' as invoice number, '2024-03-15' as date, recognizes table structure, associates Widgets with $20 total, computes subtotal, understands tax calculation, extracts total $37.80. Can answer 'What's the total?' or 'How many Widgets?' This understanding enables automation.

Handling images in RAG with multimodal models requires strategies to make visual information searchable and usable. Approaches range from captioning (convert images to text) to multimodal embeddings (direct image search) to hybrid systems.

Strategies: 1) Captioning - use VLM to generate text descriptions of images, store captions as text chunks in vector DB. Queries search captions. Simple, works with text-only RAG, but loses fine-grained visual detail. 2) Multimodal embeddings - use models like CLIP to embed images directly, store image embeddings in vector DB. Queries (text) embedded with same model, retrieve relevant images. Can return images themselves. 3) Hybrid - store both captions and image embeddings; use text search for initial retrieval, then re-rank with visual similarity. 4) Visual QA in generation - after retrieving images, pass them directly to multimodal LLM for answering questions about them. 5) Chunking images - for documents with images, treat each image as separate chunk with metadata.

Many documents contain crucial visual information - diagrams, charts, photos. Ignoring them loses information; handling them poorly (e.g., poor captions) also loses information. Choosing right strategy depends on queries: if users ask about visual details, need image embeddings or direct VLM; if queries are conceptual, captions may suffice.

Technical manual with diagrams. User asks 'How do I connect part A to part B?' Caption-only: 'Diagram showing connection of part A to part B' - helps but lacks detail. Image embedding: retrieves diagram, multimodal LLM views diagram and answers: 'Align the tab on part A with slot on part B, then push until it clicks.' This detailed visual understanding requires full multimodal RAG. For simple 'what does this diagram show?', caption sufficient.

Multimodal models are significantly more expensive than text-only models due to larger model sizes, higher computational requirements for processing images, and increased token counts (images consume many tokens). Understanding these costs is essential for budgeting and optimization.

Cost factors: 1) Model size - multimodal models are larger (vision encoder + LLM), increasing inference cost. 2) Image tokenization - images are converted to hundreds or thousands of tokens (e.g., GPT-4V uses 85x85 patches, ~4k tokens per image). Each token costs. 3) API pricing - multimodal APIs charge more (OpenAI GPT-4V ~$0.01 per image + text tokens). 4) Latency - processing images takes longer, increasing compute time (cost for self-hosted). 5) Storage - storing image embeddings or images themselves adds cost. 6) Pre-processing - OCR, captioning add steps and cost. 7) Volume - at scale, costs multiply.

A seemingly cheap multimodal API call can become expensive at scale. 1M queries with images at $0.01/image = $10,000. Text-only might be $1,000. This affects architecture decisions: maybe caption images once and store text, rather than process images per query. For production, cost optimization is critical - choose right modality mix.

Document processing pipeline with 100,000 documents/month, each with 5 images. Option A: process each image per query (10 queries/document) = 5M image API calls × $0.01 = $50,000/month. Option B: pre-caption all images once (500,000 captions × $0.01 = $5,000 one-time), then use text-only queries ($1,000/month). Option B saves $44,000/month after first month. This cost consideration drives architecture.

Audio understanding is the ability of AI models to process and interpret audio content - speech, music, sounds, and their meanings. This includes speech recognition, speaker identification, sound event detection, and audio-text understanding (e.g., answering questions about audio).

Audio models use: 1) Spectrograms - convert audio to visual representations (time-frequency images) processed by CNNs or transformers. 2) Waveform modeling - directly process raw audio with specialized architectures. 3) Speech recognition (ASR) - transcribe speech to text (Whisper). 4) Audio-text models - like CLAP (contrastive language-audio pretraining) that align audio and text embeddings. Multimodal models like Gemini and GPT-4o can process audio directly, understanding both speech and non-speech sounds. They can answer questions about audio content, summarize meetings, identify sounds (dog barking, music genre).

Audio is everywhere - meetings, calls, music, environmental sounds. Audio understanding enables: meeting transcription and summarization, voice assistants, content moderation (detecting harmful audio), accessibility (describing sounds for deaf), and media search (find podcasts about topic). It brings AI's language capabilities to the auditory world.

User uploads meeting recording and asks 'What were the action items?' Audio model: transcribes speech (Whisper), identifies speakers, extracts action items (tasks with assignees), summarizes. Responds: 'Action items: John to update the Q3 report by Friday; Sarah to schedule follow-up meeting.' This would be impossible without audio understanding. Text-only can't access audio content.

Whisper is OpenAI's automatic speech recognition (ASR) model that transcribes audio to text with high accuracy across multiple languages. It's widely used in multimodal pipelines to convert speech into text that can then be processed by LLMs or other models.

Whisper is a transformer model trained on 680,000 hours of multilingual audio data. It processes 30-second audio chunks, outputting transcribed text with timestamps. It supports: 1) Multilingual transcription (100+ languages). 2) Translation to English. 3) Voice activity detection. 4) Punctuation and casing. In multimodal pipelines, Whisper often serves as the 'audio-to-text' module: audio → Whisper → text → LLM for reasoning, summarization, QA. Some pipelines also use Whisper embeddings directly.

Whisper democratized high-quality speech recognition. It's open-source, runs locally, and works remarkably well even on challenging audio (accents, background noise). In multimodal systems, it's the go-to for adding audio understanding. Combined with LLMs, it enables meeting summarization, voice-based QA, and audio content analysis at scale.

Customer service call analysis pipeline: 1) Whisper transcribes call audio to text. 2) Text passed to LLM with prompt: 'Summarize this call, identify issues, and suggest improvements.' 3) LLM outputs structured summary, sentiment analysis, and recommendations. This pipeline processes thousands of calls automatically, extracting insights. Without Whisper, audio content inaccessible.

A text-to-image model generates images from textual descriptions (e.g., DALL-E, Stable Diffusion). A vision-language model (VLM) understands and reasons about images, outputting text. They are complementary: one creates visuals from language, the other creates language from visuals.

Text-to-image: uses diffusion models or transformers trained on image-text pairs to learn the mapping from text descriptions to image pixels. Given a prompt, it generates a novel image matching the description. Focuses on image generation, not understanding. Vision-language model: understands images, answers questions, describes content. May use similar training data but task is reversed - image to text. Some models (like Gemini) are both, capable of understanding and generating images.

Different applications need different capabilities. For creative work (design, marketing), text-to-image is essential. For analysis (QA, automation), VLM is essential. Understanding the distinction prevents using wrong model type. Some pipelines combine both: VLM analyzes an image, then text-to-image modifies it based on analysis.

Designer: 'Create a logo for a tech startup called Nexus with blue and green colors.' Text-to-image model generates logo concepts. Different task: 'What logo does this company have?' VLM analyzes image and describes it. They serve different purposes. A unified multimodal model could do both: understand existing logos and generate new ones.

Multimodal models can struggle with ambiguous or misleading images - optical illusions, abstract art, images with multiple interpretations, or deliberately deceptive visuals. Their handling depends on training data and reasoning capabilities, often revealing limitations.

Challenges: 1) Optical illusions - models may confidently give wrong interpretation because they're trained on realistic images. 2) Abstract art - may over-interpret or miss intended meaning. 3) Adversarial examples - small perturbations that fool humans? Actually fool models more easily. 4) Multiple interpretations - may pick one without acknowledging ambiguity. 5) Misleading context - image with false caption may mislead. Models typically treat images as factual, lacking human ability to recognize ambiguity or deception. They may hallucinate plausible but incorrect interpretations.

In high-stakes applications (medical, security), misinterpretation can have serious consequences. Understanding limitations helps design appropriate safeguards: human review for ambiguous cases, confidence scoring, and explicit acknowledgment of uncertainty. As models improve, handling ambiguity remains a challenge.

Optical illusion image that can be seen as both rabbit and duck. VLM asked 'What animal is this?' might confidently say 'rabbit' or 'duck', not both. Human would note ambiguity. In medical imaging, ambiguous findings require radiologist review - models should express uncertainty, not give false confidence. Handling ambiguity gracefully (saying 'this could be X or Y, further analysis needed') is a key area for improvement.

Multimodal embeddings are vector representations that map different modalities (text, images, audio) into a shared semantic space, where similar concepts are close regardless of modality. This enables cross-modal search: searching images with text, finding text that describes an image, or finding audio matching a concept.

Models like CLIP (image-text), CLAP (audio-text), and ImageBind (multiple modalities) are trained on paired data to create aligned embeddings. For any input (image, text, audio), the model produces a vector. Similarity (cosine) between vectors indicates semantic relatedness across modalities. Applications: 1) Text-to-image search - embed query text, find images with similar embeddings. 2) Image-to-text search - embed image, find relevant text. 3) Multimodal clustering - group related items across modalities. 4) Zero-shot classification - classify images by text prompts.

Cross-modal search makes all content searchable with any modality. Users can find images by describing them, find videos by their audio, or find products by uploading photos. This enables powerful search experiences and content discovery. It's the foundation of modern multimodal retrieval.

E-commerce search: user uploads photo of a dress they like. System embeds image, finds visually similar products in catalog (image-to-image). Also finds text descriptions matching style (image-to-text). User can then refine by text 'in blue'. All powered by multimodal embeddings in shared space. This creates seamless cross-modal discovery.

Multimodal LLMs are most valuable in products where users need to interact with visual, auditory, or mixed-media content using natural language. They excel at tasks that combine understanding across modalities and require reasoning.

High-value use cases: 1) Visual search and discovery - search by image, find similar products, identify objects. 2) Document processing - extract data from invoices, forms, contracts; answer questions about documents. 3) Accessibility - describe images for blind users, transcribe and summarize meetings for deaf. 4) Education - explain diagrams, answer questions about educational images. 5) Customer support - analyze screenshots of issues, guide users through visual steps. 6) Content moderation - understand images in context of text, detect harmful content. 7) Medical imaging - assist in analyzing X-rays, MRIs with reports. 8) E-commerce - virtual try-on, product recommendations from images. 9) Creative tools - generate and edit images based on conversations. 10) Robotics - understand visual scenes for navigation and manipulation.

These use cases address real user needs that text-only AI cannot. They create new product categories and enhance existing ones. For businesses, multimodal capabilities can differentiate products, improve user experience, and unlock automation in previously manual processes.

Home improvement app with multimodal AI: user takes photo of broken faucet, asks 'How do I fix this?' AI identifies faucet model, provides step-by-step repair instructions with annotated images, can even order replacement parts. This combines visual recognition, knowledge base, and e-commerce in one seamless experience. Text-only couldn't identify the faucet; image-only couldn't provide instructions. Multimodal makes it possible.