The case for on-device LLMs
Cloud models are convenient until the data is confidential. Why I keep shipping apps where the model runs entirely on your machine.
Every few weeks someone needs to translate a contract, redact a medical record, or summarize an internal document — and absolutely cannot paste it into a cloud API. That constraint is where on-device models shine.
Small models got good enough
A 1–4B parameter model, quantized and accelerated with Metal or MLX, is now genuinely useful for translation, extraction and classification. It won’t write your novel, but for narrow, well-shaped tasks it’s fast, private and free to run.
The hard part isn’t the model
In Bobr Translate, the LLM is the easy bit. The real work is surgically replacing text inside Office Open XML while preserving every font, table and chart — and packaging it all into a sandboxed, notarized Mac app. On-device means the whole pipeline has to be local, not just inference.
Privacy as a feature, not a checkbox
When nothing leaves the machine, “is this safe?” stops being a policy question. That’s a different kind of trust — and for a lot of users, it’s the only acceptable answer.