Back to the work
On-device AI · Builder

ReceiptLens.

An offline iPhone prototype that reads receipts, documents, and screenshots by running the MiniCPM-V 4.6 vision-language model fully on-device with llama.cpp, no server, no upload. A SwiftUI app wrapping an on-device multimodal engine.

Role
Builder
When
2026
Stack
Swift, SwiftUI, llama.cpp, MiniCPM-V
Scale
On-device offline VLM
GitHub ↗
ReceiptLens previewMiniCPM-V · on-device · iOS
On-deviceoffline VLM
3 modesprompt templates
SwiftUIiOS prototype
GGUFlocal model files

The problem

Reading a receipt or document with a vision-language model normally means uploading the image to a cloud API. The aim of this prototype was to keep that entirely on the phone: run a capable multimodal model (MiniCPM-V 4.6) locally so a receipt, document, or screenshot is analyzed offline, with nothing leaving the device.

What it does

  1. A SwiftUI iOS app that captures or picks an image, sends it with a mode-specific prompt to an on-device engine, and shows the model's reading, all offline.
  2. On-device inference of MiniCPM-V 4.6 (GGUF) through llama.cpp and its multimodal mtmd layer, wired into Swift via a native Objective-C++ bridge adapted from OpenBMB's official iOS demo.
  3. Three analysis modes, receipt, document, and screen, each with its own prompt template, plus an in-app model downloader that fetches the GGUF model files on first run.
  4. A local history of scans and an image file store, so past readings are kept on-device without any backend.

Impact

  • Fully offline multimodal reading: the image and the model both stay on the phone, which is the point for receipts and documents.
  • Demonstrates that a vision-language model, not just a text model, can run on-device through llama.cpp's mtmd path.
  • An honest prototype: the SwiftUI app, model management, camera flow, and engine wiring are the original work; the low-level native bridge is adapted from OpenBMB's demo and llama.cpp ships as a vendored xcframework.