Add FoundationModelsFake, a nested Swift package that fakes Apple's
Foundation Models framework, so that GeminiLanguageModel (GLM) can build
and run its unit tests on Linux and on OS versions without the real
framework. It's used only for development and testing.
- GLM depends on the fake through a `FoundationModelsFake` package
trait, which is on by default where Darwin isn't available (e.g.,
Linux). With the trait off, the fake isn't built. The root
`Package.swift` and podspecs can't reach it.
- The fake mirrors Apple's public API and availability in the 27.2
SDKs, including `@Generable` macro expansions. It omits
PrivateCloudComputeLanguageModel, OCRTool, BarcodeReaderTool, the
adapter and feedback APIs, and deprecated APIs.
SystemLanguageModel is always unavailable. Below Swift 6.4 it
compiles to an empty module.
- Decoding rejects transcript images with `file:` URLs and schemas
whose references never resolve, instead of reading local files or
overflowing the stack.
- Without ImageIO, GLM passes PNG, JPEG, WebP, HEIC, and HEIF images
through to Gemini after checking their signatures, and rejects
images that would need re-encoding.
- Add CI jobs that run both packages' tests against the fake: one on
Linux in a digest-pinned `swift:6.4.2` container, and one on macOS
with Xcode 27.2. Skip `sdk.ai` for changes that only touch the fake.
Declare the vision capability in GeminiLanguageModel and translate
prompt image attachments into Gemini inlineData parts. Previously,
any attachment segment in a transcript threw an error.
Key changes and implementation details:
- Pass-through optimization: PNG, JPEG, WebP, HEIC, and HEIF file URLs
that are already upright are sent unchanged without re-encoding.
Multi-image files are only passed through if the primary image is
the first image.
- Re-encoding and normalization: All other image sources (CGImage,
CIImage, CVPixelBuffer, other formats, or rotated files) are rotated
to display orientation and encoded as JPEG (quality 0.9, matching
Transcript Codable conformance). Grayscale, extended-range, and HDR
images are converted to sRGB, compositing transparency over black.
- Platform portability: Re-encoding relies solely on CoreGraphics,
ImageIO, and UniformTypeIdentifiers to maintain watchOS support.
- Scoping and validation: Image attachments are supported only in
prompt entries. Non-file URLs are rejected by scheme to avoid
leaking tokens or dumping long payloads.
- Ordering: Prompt segments retain their order, with adjacent text runs
merged into single text parts.
- Tests: Add extensive unit and integration tests covering formats,
orientations, color spaces, and backend image understanding.
Andrew Heard