Latency is the whole product

There is a threshold, somewhere around a couple of hundred milliseconds, below which a device feels like an extension of your thinking and above which it feels like a thing you are operating. Everything about a carried computer — which interactions people are willing to have, how often they reach for it, whether they trust it — sits on the wrong side of that line the moment a network round trip is involved.

The number that decides the interaction

Rough bands, familiar from decades of interface research rather than from any test of ours.

Under about 100 ms, a response feels instantaneous — part of the same gesture. Around 300 ms you notice, but it stays conversational. Beyond a second the interaction becomes a decision: is this worth waiting for? And past a few seconds, in a carried context, people simply stop.

Those bands are not about patience. They are about whether the device participates in what you are doing or interrupts it. A voice note captured and structured in the time it takes to put your hand down is a different product from one that shows a spinner.

What a round trip actually costs

The model’s inference time is often the smaller half. A hosted call also pays for radio wake-up from an idle state, connection setup, the request itself, queueing behind other users, and the return path — with a tail that is far worse than the average, and a floor you cannot optimise below.

Then there is the case where there is no network at all: a basement, a warehouse aisle, a train, a lift, a rural site. A carried device spends real hours there. Any capability that depends on a round trip is a capability that vanishes exactly when someone is somewhere inconvenient, which is disproportionately when it was needed.

The behavioural consequence

Here is what changes when the response is local and immediate: people use it for small things.

Marking a moment. Asking what the last decision on this was. Capturing a thought mid-sentence. Checking one number before walking into a room. Each of those is individually trivial and collectively the entire value of a carried assistant — and each is exactly the kind of interaction that a one-second delay kills, because the thing you wanted was not important enough to wait for.

An assistant that only gets used for important things is not an assistant. It is an app.

Designing for the fast path

Three principles we keep coming back to.

The common path never leaves the device. Capture, transcription, classification, the day’s context, and answers to questions you have asked before — all local, all under the threshold, always available.

Slow work announces itself. If something needs a bigger model — on your home node, or further away when you allow it — the device says so and continues doing something useful. The failure mode to avoid is uniform mediocrity, where everything takes 800 ms and nothing feels immediate.

Degradation is graceful and explicit. No network means fewer capabilities, not a broken device — and the user should be able to tell which mode they are in without checking a settings screen.

Why this is the argument for on-device

Privacy is the reason people cite for local inference and latency is the reason they keep using it. Both are true, and the second is what shows up in the first week.

A device that answers immediately becomes part of how you work. One that answers in a second and a half becomes a thing you occasionally remember to reach for. Same model quality, same features, entirely different product — decided by where the computation happens.