Offline AI Models
An offline AI model is a set of files your device can use without a server connection. Download a compatible model while online, keep its required companion files together, and run it through a local app.
Last updated:
Start with the app you will use
A model download has to match the software that will open it. If your app has a catalog, pick a model there first. If you plan to bring your own file, check the app’s supported architectures before downloading the weights.
- Loci is our pick for a first local chat. Choose a model the app recommends and download it there, without picking individual weight files. Once it is on the device, typed chat works offline and is free. Web search needs a connection.
- LM Studio gives you a desktop model browser and lets you load compatible files you already have. Get the runtime and model before you disconnect.
- Ollama suits a script or editor that needs a local API. Pull a local model while online and use its name in your requests.
- llama.cpp suits developers who want to open a specific GGUF file and set the context length or GPU offload themselves.
Match the format to the runtime
You may find several downloads for the same model. They can contain different weight precision, companion files or packaging. Follow your app’s supported format before choosing a file.
| Term | What you download | Where it belongs |
|---|---|---|
| GGUF | A container holding model weights and metadata, often in a quantized version. | llama.cpp and compatible apps. Check support for the architecture inside the file. |
| MLX package | A compatible weight conversion with configuration and tokenizer files. | MLX-based runtimes on Apple Silicon. MLX names the framework. Get the complete package your app expects. |
| Safetensors | Tensor files that hold weights. The repository may contain several shards. | A supported runtime and its matching model configuration. The filename alone does not identify the complete runtime package. |
A file with a familiar extension can still be incompatible. Architecture support arrives in runtime updates, and model authors can change the required prompt format. Use the instructions for the exact release.
Read the quantization label
On Qwen’s download page, you can choose qwen2.5-1.5b-instruct-q4_k_m.gguf. The name gives you the model family, its size class and its instruction tuning. The final two parts identify the quantization recipe and file format.
Quantization stores the model’s weights with less precision. In this release, the Q4_K_M file takes 1.12 GB on disk and the FP16 file takes 3.56 GB. Try the smaller file on a task whose answer you know before deciding how much quality you can give up.
- Q4_K_M
- A named GGUF quantization recipe. Different tensors can use different precisions. Use the published file size to compare downloads.
- 4-bit MLX
- A quantized package for an MLX runtime. It is not the same file as a 4-bit GGUF conversion.
- BF16 or FP16
- 16-bit weight representations. Expect larger weight files than many quantized alternatives.
GGUF’s quantization documentation lists block overhead and precision details. Some tensors keep higher precision. Use the published file size to plan storage instead of multiplying the number in the model name by four bits.
Use an instruction-tuned or chat variant when you want conversation. A base model is a different starting point, usually intended for further training or a specialized workflow.
Plan storage and memory separately
The figures below give the space each download takes on disk. Allow additional memory to run the model and hold the conversation.
| Download example | Published size | Keep in mind |
|---|---|---|
| Llama 3.2 3BOllama Q4_K_M | 2.0 GB | A text-model package. |
| Phi-4-mini 3.8BOllama Q4_K_M | 2.5 GB | The model still needs runtime and context memory. |
| Gemma 4 E2BGoogle QAT Q4_0 GGUF | 3.35 GB + 987 MB | Language weights plus a separate multimodal projector. |
Leave room on disk for the complete download and any installation work your app performs. In memory, allow space for the model, context and temporary work alongside the operating system.
Longer conversations and documents can increase memory use. MLX LM documents the effect of context-cache size and prompt processing. Start with a short context and increase it only after checking the result.
To compare the model families behind these files, read Best Local LLM Models.
Download, then check offline
Allow time to finish setup while you still have a connection. Try a fresh question after disconnecting, then repeat it after an app restart.
Choose a supported model
Use the app’s recommendation or check its model-format and architecture support. Confirm that you have enough free storage.
Finish the complete download
Keep the required weights, tokenizer and configuration together. For image or audio input, check for a companion projector or encoder.
Open it while connected
Ask a short question. Resolve missing-file messages before leaving your connection behind.
Turn off the network and try again
Disable Wi-Fi and mobile data, then ask a new question. Check the specific features you need; web search and new downloads still need internet.
Close and reopen the app
Try another local question after reopening. Keep the model files on the device for your trip or work session.
The offline chatbot guide explains which chat features can keep working without a connection.
Check who published the files
Start at the model maker’s official page. On a model hub, check the repository owner and the complete model name. Read the license for that release and look for a documented link to any third-party conversion.
A community conversion can be useful, but its publisher is different from the original model maker. Check which original release it used. Keep a record of that name, the quantization and the download source.
Hugging Face warns that loading a pickle file can execute code. Prefer formats supported by your app, download from publishers you trust, and avoid unknown setup scripts. A format or scan badge alone cannot establish that a repository is trustworthy.
How we chose
We checked file names and sizes against the linked model listings. For the app choices, we looked at who selects the model, which formats the app opens and whether you can run downloaded weights offline. The sizes describe the files. Memory use and response speed depend on your device and settings.
Questions and answers
What do I need to download before going offline?
Download the app or runtime, a compatible model, and any required tokenizer, configuration or projector files. Open the model once while connected. Then disconnect and try a new question, so you can catch missing files before you need the app offline.
Is GGUF a model or a file format?
GGUF is a container for model weights and metadata. Different models can use it. Your runtime must support both GGUF and the architecture inside the file; the extension alone does not establish compatibility.
Can I open an MLX model on a Windows PC?
Choose a package supported by your Windows app instead. MLX LM targets Apple Silicon. A model family may have both an MLX conversion and a GGUF conversion, but those packages are not interchangeable just because they come from the same model.
What does 4-bit quantization mean?
It stores weights at lower precision to reduce their size. The exact method matters: some tensors retain higher precision, and quantization adds metadata. A 4-bit label does not give the exact file size or guarantee the same answers as a higher-precision model.
Why does a model have a separate projector file?
Some image or audio models use a companion component to turn those inputs into a representation the language model can process. Use the files and instructions for the exact model and runtime. A text-only session may need fewer components than a multimodal session.
Does a small model file guarantee it fits in memory?
No. You also need room for the runtime, context and temporary work, alongside the operating system and other apps. Start with your app’s hardware recommendation or a small compatible model and a short context.
Where should I get model files?
Start with your app’s supported catalog or the model publisher’s official page. Check the repository owner, exact release and license. Follow a conversion link documented by the publisher or runtime. Do not run an unknown script just because it appears beside the weights.