mirror of
https://github.com/firebase/firebase-ios-sdk
synced 2026-11-09 22:36:01 +05:45
Andrew Heard
b82cdca747
Declare the vision capability in GeminiLanguageModel and translate prompt image attachments into Gemini inlineData parts. Previously, any attachment segment in a transcript threw an error. Key changes and implementation details: - Pass-through optimization: PNG, JPEG, WebP, HEIC, and HEIF file URLs that are already upright are sent unchanged without re-encoding. Multi-image files are only passed through if the primary image is the first image. - Re-encoding and normalization: All other image sources (CGImage, CIImage, CVPixelBuffer, other formats, or rotated files) are rotated to display orientation and encoded as JPEG (quality 0.9, matching Transcript Codable conformance). Grayscale, extended-range, and HDR images are converted to sRGB, compositing transparency over black. - Platform portability: Re-encoding relies solely on CoreGraphics, ImageIO, and UniformTypeIdentifiers to maintain watchOS support. - Scoping and validation: Image attachments are supported only in prompt entries. Non-file URLs are rejected by scheme to avoid leaking tokens or dumping long payloads. - Ordering: Prompt segments retain their order, with adjacent text runs merged into single text parts. - Tests: Add extensive unit and integration tests covering formats, orientations, color spaces, and backend image understanding.