An on device LLM runs directly on your phone, laptop, or other personal hardware, processing prompts locally instead of sending them to a remote server. This approach offers significant benefits in privacy and speed but comes with unique technical challenges. Understanding how these models fit within the tight constraints of consumer devices is key to using them effectively.

Key takeaways

  • Local vs. Nearby: On-device LLMs run directly on your hardware (e.g., a phone), offering privacy and low latency. Edge LLMs are a broader category that can also include processing on a nearby server, like a home hub.
  • Memory is the Main Limit: The primary constraint is your device's RAM. Total memory usage includes not just the model's weights but also the growing KV cache for conversation context, runtime overhead, and the operating system (On-device and Edge LLMs: A Systems Perspective).
  • Size Reduction is Crucial: Quantization is a technique that reduces model size by using lower-precision numbers (e.g., 4-bit instead of 16-bit) to make the model fit on a device. This can sometimes impact the model's quality or speed.
  • Efficient Tools are Essential: Technologies like the llama.cpp inference engine and the GGUF model format are specifically designed for efficient local execution without requiring heavy software dependencies (llama.cpp community integration).
  • Hardware Acceleration Isn't Always Faster: Using a dedicated accelerator like a GPU or NPU doesn't guarantee better performance. In some cases, efficient CPU-only execution can be faster due to memory access patterns. One study on an iPhone 15 Pro found a CPU-only setup generated 17 tokens per second, outperforming a GPU-accelerated configuration that reached 12.8 tokens per second (When CPUs Outperform for On-Device LLM Inference).

What is an On-Device LLM?

An on-device LLM is a large language model that performs all its computations locally on a user's hardware, such as a smartphone or laptop, without needing an internet connection to a remote server. This self-contained approach contrasts with cloud-based models that process prompts in large data centers. The goal is to provide fast, private, and offline-capable AI assistance by bringing the model directly to the user.

This is a subcategory of "edge inference." While on-device LLMs run on the end-user's device, an edge LLM could also run on a nearby server, like a gateway in a smart factory or a local server in a retail store. The core idea for both is to move computation closer to the user to reduce latency and data transmission.

The main benefits of running an LLM on your device are:

  • Privacy: Your data and prompts are not sent to a third-party server.
  • Low Latency: Responses are generated almost instantly, without network delays.
  • Offline Functionality: The model works even without an internet connection.
  • Cost: No per-token fees for inference, though there is an upfront hardware cost.

However, this approach faces significant challenges. Mobile devices and laptops have limited memory, processing power, and battery life. This means only smaller, more compact models can be used, which may not have the same capabilities as their much larger, cloud-based counterparts.

How Does On-Device LLM Inference Actually Work?

Running an on-device LLM involves a multi-step process designed to manage limited resources efficiently, from loading a compressed model into memory to generating a response one token at a time. The process relies on specialized runtimes and model formats to make inference possible on consumer hardware, turning it into a complex systems problem (On-device and Edge LLMs: A Systems Perspective).

Here is a step-by-step breakdown:

  1. Model Selection and Compression: A base model is chosen that is small enough to be a candidate for on-device use (e.g., under 7 billion parameters). It then undergoes quantization. Quantization is the process of reducing the numerical precision of the model's weights. For instance, converting 16-bit floating-point numbers to 4-bit integers drastically reduces the model's size and memory footprint.

  2. Packaging and Loading: The quantized model is saved into a special file format like GGUF. A GGUF file acts as a container, bundling the model weights, tokenizer information, and other essential metadata. An efficient runtime like llama.cpp loads this file. It often uses a technique called memory-mapping, which allows the OS to load parts of the model into RAM as needed, reducing initial memory overhead (GGUF documentation).

  3. Prefill (Processing the Prompt): When you enter a prompt, the runtime first tokenizes your text, converting words into numerical IDs. It then performs a single, intensive "forward pass" through the model to process all prompt tokens at once. This initial step calculates the attention state, known as the KV cache, for your entire prompt.

  4. Generation (Creating the Response): The model generates the response one token at a time. In each step, it uses all its weights and the entire KV cache (which now contains the state for the prompt and all previously generated tokens) to predict the next token.

  5. Managing Context: With each new token generated, the KV cache grows, consuming more RAM. This is a critical challenge for on-device models, as a long conversation can quickly exhaust a device's memory (On-device and Edge LLMs: A Systems Perspective). The runtime must manage this cache efficiently to sustain a conversation.

  6. Hardware Execution: The runtime intelligently schedules all these calculations across the device's available hardware: the CPU, GPU (graphics processing unit), or NPU (neural processing unit). The optimal allocation depends on the model, device, and specific task.

How Much RAM Does a Mobile LLM Really Need?

The total RAM an on-device LLM needs is more than just the model's file size. You must account for the model weights, the dynamic KV cache that stores conversation context, and overhead from the app and operating system. A good rule of thumb is: Total RAM Needed = Model Size + KV Cache Size + Runtime & OS Overhead. Failing to account for all three is a common pitfall.

Let's walk through an example for a 3-billion-parameter model on a smartphone.

Illustrative Example: 3-Billion-Parameter Model

  1. Model Size (Weights): The model's parameters are stored as numbers. The memory required depends on their precision (quantization).

    • Unquantized (16-bit float): Each parameter takes 2 bytes. 3,000,000,000 parameters × 2 bytes/parameter = 6,000,000,000 bytes = 6 GB
    • Quantized (4-bit): After quantization, each parameter takes roughly 0.5 to 0.6 bytes on average (including metadata). 3,000,000,000 parameters × ~0.56 bytes/parameter ≈ 1.7 GB Already, quantization makes the difference between fitting on a high-end phone and not fitting at all. This 1.7 GB must be available in your device's RAM.
  2. KV Cache Size (Context): This memory is used to keep track of the conversation so the model doesn't forget what was said. Its size grows with the length of the prompt and the generated response.

    • The exact formula is complex, but for a typical 3B model, a 4096-token context window can easily consume over 1 GB of RAM.
  3. Runtime & OS Overhead: The application itself, the llama.cpp runtime, and the phone's operating system all require memory.

    • This can vary widely, but it's safe to budget 0.5 GB to 1 GB for this overhead.

Putting It All Together:

For our 4-bit quantized 3B model with a 4096-token context: 1.7 GB (Model) + 1.0 GB (KV Cache) + 0.8 GB (Overhead) = 3.5 GB of required free RAM.

An 8 GB smartphone, which may only have 3-4 GB of free RAM after the OS and background apps, would be at its absolute limit. This makes running even a small LLM a significant challenge on many current devices.

The table below gives rough estimates for different model sizes, assuming 4-bit quantization and a 4096-token context.

Model Size (Parameters) Quantized Model Size Est. KV Cache Size Est. Total RAM Needed Feasible On
1 Billion ~0.6 GB ~0.5 GB ~1.9 GB+ Most modern smartphones/laptops
3 Billion ~1.7 GB ~1.0 GB ~3.5 GB+ High-end phones, standard laptops
7 Billion ~4.0 GB ~1.5 GB ~6.2 GB+ Laptops with 16GB+ RAM, very high-end phones

Note: These are simplified estimates. Actual memory usage depends on the exact model architecture, quantization method, and runtime.

On-Device vs. Edge vs. Cloud: Which is Right?

Choosing where to run an LLM involves trading off capability against privacy, latency, and cost. On-device offers the most privacy, the cloud offers the most power, and edge computing sits in between. A hybrid approach, where simple tasks run locally and complex ones are sent to the cloud, is often the most practical solution.

The best choice depends entirely on your application's requirements.

Feature On-Device Inference Edge Inference Cloud Inference
Best For Instant feedback, high privacy, offline tasks (e.g., text suggestions). Real-time shared tasks in a local environment (e.g., factory automation, smart home). Complex reasoning, large-scale data analysis, accessing the most powerful models.
Latency Lowest (milliseconds). Low (tens of milliseconds). Highest (hundreds of milliseconds to seconds).
Privacy Highest. Prompts do not leave the user's device. High. Data stays within a local network. Lowest. Prompts are sent to a third-party service.
Offline Access Fully functional without an internet connection. Functional within the local network, but may depend on a gateway. Not available. Requires a stable internet connection.
Model Capability Limited to small, quantized models (typically < 7B parameters). Can use medium-sized models on more powerful edge hardware. Access to the largest and most capable models available (100B+ parameters).
Hardware Mgmt. None. Uses the end-user's device. Requires purchasing and maintaining local edge servers. None. Managed entirely by the cloud provider.

Why Isn't My GPU or NPU Making Inference Faster?

Using a powerful GPU or NPU doesn't always speed up on-device LLM inference; sometimes, the CPU is faster. This counterintuitive result happens because LLM inference is often memory-bound, not compute-bound. The bottleneck is the speed at which the model's weights can be moved from RAM to the processor, not the speed of the calculations themselves.

CPUs are designed for general-purpose tasks and typically have extremely fast, direct access to the main system RAM. In contrast, specialized accelerators like GPUs and NPUs have their own memory systems. To use them, data (the model weights and KV cache) must be copied from system RAM to the accelerator's memory. This data transfer introduces overhead.

For smaller models or certain workloads, the time spent copying data to the GPU/NPU can be greater than the time saved by its faster processing.

A 2025 study demonstrated this on an iPhone 15 Pro with a 1-billion-parameter model (When CPUs Outperform for On-Device LLM Inference). The researchers found:

  • CPU-only (2 threads): Achieved 17 tokens per second.
  • GPU-accelerated: Achieved 12.8 tokens per second.

In this case, the efficiency of the CPU's direct memory access outweighed the raw computational power of the GPU. The optimal hardware for inference is not a theoretical absolute; it depends on the specific device, model, runtime, and the workload being executed.

Common Mistakes and When Not to Use On-Device LLMs

While powerful, on-device LLMs are not a universal solution. Misunderstanding their limitations can lead to poor performance and a frustrating user experience.

Common Mistakes to Avoid

  • Ignoring the KV Cache: Budgeting memory just for the model weights is a frequent error. The KV cache for context can consume gigabytes of RAM in a long conversation, leading to crashes or slowdowns.
  • Assuming Total Privacy: Local inference is private, but the application using the model may not be. Telemetry, crash reports, or cloud-sync features can still send your data over the network.
  • Forgetting About Battery and Heat: Running an LLM is computationally intensive. On a mobile device, this can drain the battery quickly and cause the device to heat up, leading to thermal throttling where the processor slows down to cool off.
  • Relying on Parameter Count Alone: A 3B model isn't always better than a 1B model. Performance depends on quantization, training data, and architecture. Evaluating accuracy, latency, and memory usage together is essential (When CPUs Outperform for On-Device LLM Inference).

When to Use a Cloud LLM Instead

You should probably avoid an on-device approach and use a cloud-based model when your application:

  • Requires State-of-the-Art Performance: If you need the highest quality reasoning, creativity, or factual accuracy, the largest cloud models are unbeatable.
  • Needs Access to Real-Time Information: On-device models are static and have no knowledge of events after their training date. Cloud models can be connected to search engines for up-to-date answers.
  • Serves a Wide Range of Devices: If you can't guarantee your users have high-end hardware with enough RAM and processing power, a cloud-based solution provides a consistent experience for everyone.
  • Demands Frequent and Easy Model Updates: Updating a model in a cloud backend is instant for all users. Rolling out updates to an on-device model requires users to download a large file, which can be slow and inconsistent.

What to learn next

Sources and further reading