In AI, quantization means shrinking a model by storing its numbers with less precision, so it needs less memory. It is what lets big models run on ordinary laptops and phones, at the cost of a little quality.
Think of prices in a shop. You could write £4.9937 on every label, but £4.99 is nearly the same and takes less room. Quantization does that to the billions of numbers inside a model: it rounds them off. The model gets much smaller and still behaves almost the same, though some fine detail is lost.
In a sentence: “The full model wouldn’t fit on my laptop, so I downloaded a 4-bit quantization of it instead.”
What it means in AI
A model is a huge list of numbers; quantization stores each one more roughly. Those numbers, called weights, are what the model learned in training. IBM’s general definition: quantization is “reducing the precision of a digital signal”, usually from a more precise format to a less precise one.1
The goal is to save memory without changing the answers much. Hugging Face, a site where AI models are shared, says quantization aims at “reducing the precision of weights while trying to keep the model’s inference results as accurate as possible”. Inference simply means the model producing an answer.2
A simple example
The “Q4” in a download name means 4-bit: a heavily shrunk copy. When you pick a model in a home app like LM Studio, you often see several versions of the same model with labels like Q4 or Q8. Smaller numbers mean more squeezing. LM Studio says quantization shrinks the file “while giving up some degree of quality”, and suggests choosing “a 4-bit option or higher” if your computer can run it.3
The saving is large. In one Hugging Face test, a model that needed about 32 GB of graphics memory needed about 15 GB at 8-bit and just over 9 GB at 4-bit.2
Why it matters to you
- It decides whether a model fits on your machine. IBM says quantized models can be used “on laptops, tablets and smartphones.”1 VRAM meaning explains the memory limit it gets around.
- Some quality is lost. IBM calls the loss “quantization error”.1 The more you shrink, the more you risk odd answers. If a small file gives poor results, try a less squeezed version.
- Speed can go either way. IBM says smaller numbers can be worked out faster. Hugging Face found that with the method it tested, answers came a little slower, and sums up the trade: quantization “trades improved memory efficiency against accuracy and in some cases inference time.”2
Often confused with
| Word | How it differs |
|---|---|
| Zipping a file | Differs: zipping shrinks a file and gives back the exact original. Quantization throws a little detail away for good. |
| A smaller model | Differs: a small model has fewer numbers in the first place. A quantized model has the same number of them, each stored more roughly. |
| Fine-tuning | Differs: fine-tuning teaches a model new behaviour. Quantization only changes how it is stored. See fine-tuning meaning. |
Questions people ask
Does quantization make a model worse?
A little. Lower precision means some lost accuracy, which IBM calls “quantization error”, and Hugging Face says quantization “trades improved memory efficiency against accuracy”.12
What does “4-bit” or “8-bit” mean?
It is how many bits each of the model’s numbers is stored in; fewer bits, smaller file. In Hugging Face’s example, one model needed 32 GB of graphics memory, 15 GB at 8-bit, and “just a bit over 9GB” at 4-bit.2 LM Studio suggests “a 4-bit option or higher if your machine is capable enough”.3
Why would you quantize a model?
To run it on ordinary hardware. IBM says quantization makes it possible “to utilize these quantized models within real-world applications on laptops, tablets and smartphones.”1
Learn more on this site
- Run AI on your own computer — choosing a download option in LM Studio and Ollama.
- Local or cloud — what you give up by running smaller copies at home.
- Open weights meaning — the kind of model you are allowed to download and shrink.
- The AI glossary and AI for beginners — the basics, in plain words.
Related on this site: Small language model (SLM) meaning
QUICK CHECK · 3 QUESTIONS
Pick an answer to see if you got it. Nothing is recorded.
1. What does quantization do to an AI model?
- Stores its numbers with less precision so it needs less memory
- Teaches it new skills from fresh examples
- Gives it more numbers so it knows more
Show the answer
Answer: Stores its numbers with less precision so it needs less memory. Like writing a price as 4.99 instead of a longer figure, quantization rounds off the model’s numbers so it gets much smaller.
2. What does a label like “Q4” on a model download mean?
- It is the fourth version of the model
- It is a 4-bit, heavily shrunk copy
- It only works on four computers
Show the answer
Answer: It is a 4-bit, heavily shrunk copy. Smaller numbers in these labels mean more squeezing and a smaller file.
3. How does quantization differ from zipping a file?
- They work exactly the same way
- Zipping throws detail away, quantization keeps everything
- Zipping gives back the exact original, while quantization loses a little detail for good
Show the answer
Answer: Zipping gives back the exact original, while quantization loses a little detail for good. Some quality is lost with quantization, so if a small file gives poor results, try a less squeezed version.
Related terms: GPU · Local LLM / on-device AI · NPU · Small language model (SLM) · VRAM
- 1IBM, What is Quantization?, undated, read at source 1 Oct 2026“Quantization is the process of reducing the precision of a digital signal, typically from a higher-precision format to a lower-precision format.” ↑a“it is possible to utilize these quantized models within real-world applications on laptops, tablets and smartphones.” ↑b“This effectively results in less accuracy, also referred to as quantization error.” ↑c“This effectively results in less accuracy, also referred to as quantization error.” ↑d“it is possible to utilize these quantized models within real-world applications on laptops, tablets and smartphones.” ↑e
- 2Hugging Face, Optimizing LLMs for Speed and Memory, undated documentation, read at source 1 Oct 2026“quantization schemes aim at reducing the precision of weights while trying to keep the model’s inference results as accurate as possible”. ↑a“running OctoCoder in 8-bit precision reduced the required GPU VRAM from 32G GPU VRAM to only 15GB and running the model in 4-bit precision further reduces the required GPU VRAM to just a bit over 9GB.” ↑b“model quantization trades improved memory efficiency against accuracy and in some cases inference time.” IBM, What is Quantization?, undated, read at source 1 Oct 2026: “the calculations can be completed more rapidly.” ↑c“model quantization trades improved memory efficiency against accuracy and in some cases inference time.” ↑d“running OctoCoder in 8-bit precision reduced the required GPU VRAM from 32G GPU VRAM to only 15GB and running the model in 4-bit precision further reduces the required GPU VRAM to just a bit over 9GB.” ↑e
- 3LM Studio, Download an LLM, read at source 1 Oct 2026“‘Quantization’, which roughly means compressing model files in size, while giving up some degree of quality.” and “Choose a 4-bit option or higher if your machine is capable enough for running it.” ↑a“Choose a 4-bit option or higher if your machine is capable enough for running it.” ↑b
The comparisons on this page (the shop price labels, zipping a file) are this site’s own ways of explaining, not claims from these sources.
Free printable AI cheat sheets, and AI news every week.
Sixteen one-page AI guides and checklists in one free PDF, no sign-up. And if you want to keep up, the newsletter covers what changed in AI each week, in plain English, with the sources.
SUBSCRIBE FREE ↗ GET THE CHEAT SHEET PACK