A small language model, or SLM, means a text AI built much smaller than the big chatbot models, so it needs far less memory and computing power. That is why small models can run on a phone or an ordinary laptop, even offline, though they know less and handle hard questions less well.
Think of a pocket dictionary next to a full encyclopedia set. The pocket book fits in your bag and answers everyday questions quickly, anywhere. The encyclopedia knows far more, but it needs a library to hold it. A small language model is the pocket book; a large one lives in a company’s data centre.
In a sentence: “I switched my note-taking app to a small language model, so it can now sum up my notes on my phone even without Wi-Fi.”
What the term means
The plain point: the same kind of AI as a large language model, just with far fewer parameters.
IBM, the computer company, says SLMs “are smaller in scale and scope than large language models (LLMs).” It puts their size at a few million to a few billion parameters, the numbers a model learns in training, against hundreds of billions or even trillions for LLMs.1
Makers use the word loosely. Microsoft’s Phi-3 family of small language models includes a 14-billion-parameter model, well above IBM’s “a few billion”.2 So treat “small” as a rough size label, not a fixed line.
Why small models run on phones and laptops
The plain point: a whole model has to fit in your device’s memory, and a small one does.
IBM says SLMs “require less memory and computational power”, which suits phones, other small devices and offline use.3 Microsoft, which makes the Phi family of small models, adds that keeping data on the device helps to “minimize latency and maximize privacy”; latency is the wait for an answer.2
Models are often shrunk further to fit, by storing their numbers less precisely (quantization) or training a small model to copy a big one (distillation). IBM lists both as standard methods.4
A simple example
The plain point: you may already carry one.
Apple says the model inside Apple Intelligence that runs on the device has about 3 billion parameters, and is good at jobs like summarising and rewriting. Apple is also clear what it is not: “It is not designed to be a chatbot for general world knowledge.”5 Google’s Gemini Nano does a similar job on some Android phones, such as summarising a recording with no internet connection.6
Small models are made for focused jobs on your own device. For broad knowledge or hard reasoning, a big model in the cloud usually does better.
What you give up
The plain point: smaller means less room to store facts.
IBM quotes Microsoft on its own small models: “Phi-3 models do not perform as well on factual knowledge benchmarks as the smaller model size results in less capacity to retain facts.” IBM also lists weaker performance on complex tasks and the same risk of made-up answers as big models.4 Microsoft’s researchers expect a gap with the big cloud models to remain, “because you know – the big models are going to keep making progress.”7
Often confused with
| Word | What it means | How it differs |
|---|---|---|
| SLM | Means: a language model small enough for modest hardware. | Like: the pocket dictionary. |
| LLM | Means: a large language model, usually run in a data centre. | Differs: knows more, needs far more memory. |
| Local AI | Means: any model running on your own device. | Differs: describes where it runs, not its size. Small models make it practical. |
Questions people ask
Can I run a small language model on my laptop?
Yes, if the model fits in your computer’s memory. Tools such as Ollama and LM Studio run models on your own machine, and LM Studio explains that loading one means making room for it in “your computer's RAM”.8 Run AI on your own computer lists the memory each needs.
Is a small language model worse than a large one?
At broad knowledge and hard problems, usually: IBM quotes Microsoft saying its small models hold fewer facts, and lists the same risk of made-up answers as big models.4 For a focused job on your own device, a small model can be the better fit, because it needs less memory and can work offline.3 Microsoft expects a gap with the biggest models to remain.7
Learn more on this site
- Running one yourself: run AI on your own computer.
- Where AI runs: local AI meaning and local or cloud?
- Making models smaller: quantization meaning and distillation meaning.
- The words: LLM meaning and the AI glossary.
QUICK CHECK · 3 QUESTIONS
Pick an answer to see if you got it. Nothing is recorded.
1. Why can a small language model run on a phone or laptop?
- It fetches every answer from the internet
- It only accepts very short questions
- It needs far less memory and computing power
Show the answer
Answer: It needs far less memory and computing power. The page says small models need much less memory and computing power, so the whole model can fit on your own device.
2. What do you usually give up with a small model?
- Broad knowledge, since there is less room to store facts
- The ability to work without Wi-Fi
- The ability to summarise or rewrite text
Show the answer
Answer: Broad knowledge, since there is less room to store facts. The page says smaller means less room to store facts, and small models do less well on hard, broad questions.
3. How should you treat the word ‘small’ in small language model?
- As an exact size set by a standards body
- As a rough size label, not a fixed line
- As a warning that the model is unsafe
Show the answer
Answer: As a rough size label, not a fixed line. The page says makers use the word loosely, so ‘small’ is a rough size label rather than a fixed limit.
Related terms: GPU · Local LLM / on-device AI · NPU · Quantization · VRAM
- 1IBM, What are small language models?, published 31 Oct 2024, updated 2 Apr 2026, read at source 5 Oct 2026“As their name implies, SLMs are smaller in scale and scope than large language models (LLMs). In terms of size, SLM parameters range from a few million to a few billion, as opposed to LLMs with hundreds of billions or even trillions of parameters.” ↑
- 2Microsoft, Tiny but mighty: The Phi-3 small language models with big potential, 23 Apr 2024, read at source 5 Oct 2026“Phi-3-small (7 billion parameters) and Phi-3-medium (14 billion parameters) will be available”. ↑a“By keeping data within the device, users can ‘minimize latency and maximize privacy’”. ↑b
- 3IBM, What are small language models?, updated 2 Apr 2026, read at source 5 Oct 2026“SLMs require less memory and computational power, making them ideal for resource-constrained environments such as edge devices and mobile apps, or even for scenarios where AI inferencing—when a model generates a response to a user’s query—must be done offline without a data network.” ↑a“SLMs require less memory and computational power, making them ideal for resource-constrained environments such as edge devices and mobile apps, or even for scenarios where AI inferencing—when a model generates a response to a user’s query—must be done offline without a data network.” ↑b
- 4IBM, What are small language models?, read at source 5 Oct 2026“Here are some common model compression methods: Pruning Quantization Low-rank factorization Knowledge distillation”. ↑a“For example, Microsoft notes that its ‘Phi-3 models do not perform as well on factual knowledge benchmarks as the smaller model size results in less capacity to retain facts.’” … “Hallucinations: Validating the results of SLMs is vital to make sure what they produce is factually correct.” ↑b“For example, Microsoft notes that its ‘Phi-3 models do not perform as well on factual knowledge benchmarks as the smaller model size results in less capacity to retain facts.’” … “Hallucinations: Validating the results of SLMs is vital to make sure what they produce is factually correct.” ↑c
- 5Apple Machine Learning Research, Updates to Apple’s On-Device and Server Foundation Language Models, 9 Jun 2025, read at source 5 Oct 2026“The ~3B language foundation model at the core of Apple Intelligence excels at a diverse range of text tasks, like summarization, entity extraction, text understanding, refinement, short dialog, generating creative content, and more. It is not designed to be a chatbot for general world knowledge.” ↑
- 6Google, Android Developers Blog, A New Foundation for AI on Android, 6 Dec 2023, read at source 5 Oct 2026“Gemini Nano enables the Pixel 8 Pro to concisely summarize content in the Recorder app, even when the phone’s network connection is offline.” ↑
- 7Microsoft, Tiny but mighty, 23 Apr 2024, read at source 5 Oct 2026“There’s still a gap between small language models and the level of intelligence that you can get from the big models on the cloud” … “And maybe there will always be a gap because you know – the big models are going to keep making progress.” ↑a“There’s still a gap between small language models and the level of intelligence that you can get from the big models on the cloud” … “And maybe there will always be a gap because you know – the big models are going to keep making progress.” ↑b
- 8LM Studio, Get started with LM Studio, read at source 23 Sep 2026 (as cited on Run AI on your own computer): “Loading a model typically means allocating memory to be able to accommodate the model's weights and other parameters in your computer's RAM.”↑
The comparison on this page (the pocket dictionary and the encyclopedia) is this site’s own way of explaining, not a claim from these sources.
Free printable AI cheat sheets, and AI news every week.
Sixteen one-page AI guides and checklists in one free PDF, no sign-up. And if you want to keep up, the newsletter covers what changed in AI each week, in plain English, with the sources.
SUBSCRIBE FREE ↗ GET THE CHEAT SHEET PACK