Local AI First: Why On-Device Intelligence Should Be Our Default
I read that (unix.foo article) about local AI needing to be the norm, and it really resonated with me. It talked about something I’ve been turning over in my head for a while now: the most powerful AI we’ve got isn’t humming away in some distant data center, rather it’s sitting right here locally on our phones and laptops.
What struck me was how we’ve ended up treating every AI feature like an excuse to build another fragile distributed system. It lives and dies by network connections, hinges on some third-party server staying online, and requires us to juggle API keys like they’re precious artifacts. When we ship user data off to these cloud AI services, we’re suddenly dealing with privacy headaches, data retention questions, and a whole layer of complexity we never signed up for.
The Brutalist Report example really made it click for me. Their iOS app uses Apple’s on-device language models for article summaries, and the result is pretty neat: AI that works completely offline, never sends a byte of user data anywhere, and doesn’t require those tedious privacy explanations we’ve all learned to ignore. This approach represents a fundamentally different way to earn user trust.
The author makes a point I completely agree with: local AI truly shines when it’s working with data the user already possesses, not when we’re trying to make it into some universal oracle of knowledge. Think about practical stuff like summarizing long emails, extracting action items from meeting notes, or automatically sorting documents into sensible categories. These are ideal tasks for on-device models. Why bother exporting data somewhere else for processing when it’s already sitting right there on the user’s device?
I also appreciated seeing those Swift examples showing how Apple’s tools have evolved. The shift from getting back raw text responses to actually getting structured data through the Generable protocol is genuinely useful. When your AI output becomes a proper type in your codebase instead of just an amorphous blob of text to parse, you suddenly gain reliability and maintainability that cloud-based approaches simply can’t match.
Of course, someone might object that “local models aren’t as smart”, and from a certain perspective, they’re technically right. But does it actually matter for most applications? Most app features don’t need a model capable of writing Shakespearean sonnets or explaining quantum chromodynamics. They need to perform one specific task well: summarizing, categorizing, extracting information, rewriting text, or normalizing data formats. For these concrete data transformation jobs, local models can be absolutely outstanding; especially when they’re working with information the user has deliberately chosen to engage with.
This perspective aligns with how I try to see software development: prioritizing user autonomy, keeping external dependencies to an absolute minimum, and treating privacy as something to design in from the very beginning rather than bolting on as an afterthought. When we choose to process AI locally, we’re making a statement about who rightfully controls user data and where that data fundamentally belongs.
That closing line from the unix.foo article has stuck with me: “Stop shipping distributed systems when you meant to ship a feature.” It’s become a kind of guiding principle for me. Local AI isn’t merely another technical alternative to consider; it represents a return to first principles about building software that genuinely respects the people who use it.
Crepi il lupo! 🐺