Quantization in practice: running a capable model on one consumer GPU

LEARN · LLMS & TRANSFORMERS

The goal: make a model fit without making it useless

Running a capable language model locally is mostly a memory-management problem.

A modern model may have enough parameters that its uncompressed weights alone exceed the VRAM on a typical consumer GPU. Loading every parameter in BF16 means storing roughly two bytes per parameter. For a 9-billion-parameter language model, that is about 18 GB of raw weight storage before counting the CUDA context, temporary buffers, attention state, tokenizer-related structures, or anything else needed to generate a token.

Quantization changes that equation by representing most weights with fewer bits.

At 4 bits per weight, the same 9 billion parameters have a theoretical raw storage requirement of about 4.5 GB:

Raw weight storage ≈ parameter count × bits per parameter ÷ 8

That does not mean a 9B model occupies exactly 4.5 GB of VRAM. Quantizers need scales and metadata, some tensors normally remain at higher precision, and generation itself consumes memory. But it changes the problem from “obviously does not fit” to “comfortably plausible on a 12 GB GPU.”

For a concrete target, we will use Qwen3.5-9B. Its current official model card describes a 9B-parameter language component with 32 language layers and a native context limit of 262,144 tokens. It is also a multimodal model, so the complete checkpoint contains more than just those language weights.

We will run it two ways:

Load the original checkpoint with Transformers and quantize its linear layers to 4-bit at load time using bitsandbytes.
Run a pre-quantized 4-bit .gguf checkpoint with llama.cpp.

The first path is ideal when your application already lives in Python. The second is attractive when you want a lean local runtime, a command-line program, or a small local model server.

The important lesson is not merely how to make the model fit. It is how to tell whether the memory you saved was worth the quality you gave up.

What “4-bit” actually means

It is tempting to think of quantization as converting every floating-point number in the model into a four-bit integer.

Real implementations are more nuanced.

A useful 4-bit scheme generally stores quantized weight values together with scaling information that lets the runtime approximately reconstruct the original numerical range during matrix multiplication. Different groups of weights may have separate scales. Some tensors remain at 16- or 32-bit precision because quantizing them provides little benefit or harms numerical behavior.

That is why “4-bit model” is best understood as a storage and computation strategy rather than a literal statement that every byte in the process has been reduced by exactly 75%.

Hugging Face’s current bitsandbytes integration exposes several controls for this. Its 4-bit configuration supports NF4, configurable compute dtypes, and nested or “double” quantization. The documentation notes that nested quantization can save an additional 0.4 bits per parameter by quantizing quantization metadata itself.

There is another distinction that matters:

Weight precision is not compute precision.

You can store weights in 4-bit form while carrying out much of the arithmetic in BF16 or FP16. That is exactly what we will do. The low-bit representation reduces memory pressure; the wider compute type gives the GPU a more practical arithmetic format.

Hardware assumptions

The bitsandbytes example below targets one NVIDIA GPU.

Current bitsandbytes documentation requires Python 3.10 or newer and PyTorch 2.4 or newer. For NF4/FP4 quantization on NVIDIA hardware, the documented minimum is compute capability 6.0. Current packages support substantially newer CUDA and GPU generations as well.

You do not need a datacenter card.

A 12 GB consumer GPU is a useful target for this exercise because the unquantized model is clearly too large while the 4-bit representation leaves enough room for realistic generation. An 8 GB card may also work with aggressive memory discipline, but context length and runtime overhead become much more important.

Before doing anything else, check the GPU:

nvidia-smi

You want to know three things:

Total VRAM.
Whether another process is already consuming several gigabytes.
Whether your NVIDIA driver is functioning before Python enters the picture.

Quantization cannot recover memory occupied by your browser, desktop compositor, another model server, or an abandoned Python process.

Path one: 4-bit loading with Transformers and bitsandbytes

Create an isolated environment

Start with a clean virtual environment so that an old Transformers or bitsandbytes installation does not quietly determine your results.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install --upgrade torch torchvision transformers accelerate bitsandbytes

Qwen’s current model documentation uses the modern Transformers multimodal loading stack with AutoProcessor and AutoModelForMultimodalLM. Keeping Transformers current is particularly important for a recent architecture such as Qwen3.5.

Verify that Python can actually see your GPU:

python - <<'PY'
import torch
import transformers
import bitsandbytes

print("PyTorch:", torch.__version__)
print("Transformers:", transformers.__version__)
print("bitsandbytes:", bitsandbytes.__version__)
print("CUDA available:", torch.cuda.is_available())

if torch.cuda.is_available():
    print("GPU:", torch.cuda.get_device_name(0))
    print("BF16 supported:", torch.cuda.is_bf16_supported())
PY

Do not continue merely because installation succeeded.

If the output says CUDA available: False, you have a PyTorch/CUDA installation problem, not a quantization problem.

Build the complete 4-bit loader

Create a file named run_bnb.py:

import time

import torch
from transformers import (
    AutoModelForMultimodalLM,
    AutoProcessor,
    BitsAndBytesConfig,
)

MODEL_ID = "Qwen/Qwen3.5-9B"
DEVICE = "cuda:0"

if not torch.cuda.is_available():
    raise SystemExit("This example requires an NVIDIA CUDA GPU.")

compute_dtype = (
    torch.bfloat16
    if torch.cuda.is_bf16_supported()
    else torch.float16
)

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)

print(f"Using compute dtype: {compute_dtype}")

processor = AutoProcessor.from_pretrained(MODEL_ID)

model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    quantization_config=quantization_config,
    device_map={"": 0},
    dtype=compute_dtype,
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": (
                    "Explain why a database index can speed up reads "
                    "but slow down writes. Use exactly five concise bullets."
                ),
            }
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    enable_thinking=False,
).to(DEVICE)

torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()

start = time.perf_counter()

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=300,
        do_sample=True,
        temperature=0.7,
        top_p=0.8,
        top_k=20,
    )

torch.cuda.synchronize()
elapsed = time.perf_counter() - start

prompt_tokens = inputs["input_ids"].shape[-1]
generated = outputs[0, prompt_tokens:]
new_tokens = generated.shape[-1]

text = processor.decode(
    generated,
    skip_special_tokens=True,
)

model_gib = model.get_memory_footprint() / (1024 ** 3)
peak_gib = torch.cuda.max_memory_allocated() / (1024 ** 3)
tokens_per_second = new_tokens / elapsed

print()
print(text)
print()
print(f"Generated tokens: {new_tokens}")
print(f"Generation time: {elapsed:.2f} s")
print(f"Generation speed: {tokens_per_second:.2f} tokens/s")
print(f"Model footprint reported by Transformers: {model_gib:.2f} GiB")
print(f"Peak PyTorch CUDA allocation during generation: {peak_gib:.2f} GiB")

Run it:

python run_bnb.py

The first run downloads the checkpoint, so model loading may be dominated by network and disk activity. Measure generation speed only after the model has finished loading.

The current Qwen3.5 model card documents the processor-based chat interface used here. It also documents disabling the model’s default thinking behavior through its chat-template configuration, which is useful when you want a short direct answer rather than reasoning-oriented output.

Why each 4-bit setting is there

The configuration is short, but none of its fields is decorative.

`load_in_4bit=True`

This tells the Transformers/bitsandbytes integration to replace supported linear layers with 4-bit quantized equivalents while loading the checkpoint.

That distinction is useful operationally: you are downloading the original model rather than maintaining a separately converted local checkpoint.

`bnb_4bit_quant_type="nf4"`

NF4 is the four-bit data type provided by bitsandbytes for this workflow. Current Hugging Face documentation exposes it directly through BitsAndBytesConfig.

Do not interpret that choice as a universal theorem that NF4 must always produce the best inference quality. Quantization behavior depends on model architecture, weight distribution, calibration strategy, kernels, and workload.

`bnb_4bit_use_double_quant=True`

Double quantization compresses some of the quantization constants themselves.

The current documentation reports an additional saving of about 0.4 bits per parameter from this nested quantization technique. On a small model that difference may seem trivial. Across billions of parameters, it becomes meaningful.

`bnb_4bit_compute_dtype=...`

This does not turn the stored weights back into BF16.

It selects a wider type for computation. BF16 is preferable when your GPU supports it; otherwise the example falls back to FP16.

This is a good illustration of why a single phrase such as “4-bit inference” hides a lot of implementation detail. Storage precision, accumulator precision, activation precision, cache precision, and individual kernels can all differ.

`device_map={"": 0}`

You will often see examples using automatic device placement.

That is convenient, but it is less useful for this experiment. Automatic placement may decide to put some modules somewhere you did not expect.

Here we deliberately request GPU zero because the question we are answering is specific: can the model run on one GPU?

If that configuration fails with an out-of-memory error, you have learned something real instead of accidentally benchmarking a mixed CPU/GPU setup.

Measure the board, not just the Python object

model.get_memory_footprint() is useful, but it is not the whole story.

Likewise, torch.cuda.max_memory_allocated() describes memory managed through PyTorch’s CUDA allocator. The GPU also contains driver state and other allocations.

While the model is running, inspect the card from another terminal:

watch -n 0.5 nvidia-smi

Look at the high-water mark during actual generation.

This matters because loading a model successfully is not the same as having enough memory to use it.

You may load at 6 or 7 GiB, then trigger an out-of-memory error after sending a long prompt because generation needs additional runtime state.

That brings us to the most common local-model mistake.

The context-window trap

Qwen3.5-9B advertises a native context length of 262,144 tokens. That number describes what the architecture supports; it is not advice to allocate that context length on a consumer GPU.

Model weights and context memory are separate parts of your VRAM budget.

Quantizing the weights solves only the first problem.

During generation, the runtime must retain state associated with the sequence. The exact memory behavior depends on the architecture and implementation, and Qwen3.5 uses a hybrid design rather than a textbook stack of identical full-attention layers. Even so, longer contexts still consume more runtime memory and computation.

For local work, start small:

4,096 tokens for simple assistants and extraction tasks.
8,192 tokens for coding sessions and moderate documents.
Increase only after measuring actual VRAM use.
Do not configure a six-figure context merely because the model card says the architecture can support it.

A local model that runs reliably at 8K is much more useful than one that crashes while theoretically supporting 262K.

Path two: a pre-quantized model with llama.cpp

The Transformers path performs quantization as part of loading the original checkpoint.

llama.cpp takes a different approach. It can consume models already converted and quantized into the .gguf file format.

The project currently supports multiple low-bit integer quantizations, CUDA acceleration for NVIDIA GPUs, and CPU/GPU hybrid execution. Its current quick-start tooling also supports fetching compatible model files directly from Hugging Face.

For our experiment, we still want one GPU rather than hybrid offloading.

Build llama.cpp with CUDA support

On an Ubuntu-like Linux system, install the build dependencies:

sudo apt update
sudo apt install -y build-essential cmake git libcurl4-openssl-dev

Clone and build the current source:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j --target llama-cli llama-server

The CUDA backend is the important part here. Current llama.cpp supports dedicated CUDA kernels for NVIDIA hardware as well as other accelerator backends.

Run a Q4_K_M build

A current Qwen3.5-9B community conversion provides a Q4_K_M variant that can be fetched using llama.cpp’s Hugging Face integration.

Run it like this:

CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-cli \
  -hf lmstudio-community/Qwen3.5-9B-GGUF:Q4_K_M \
  -ngl 99 \
  -c 8192

When the interactive prompt appears, try:

Explain why a database index can speed up reads but slow down writes. Use exactly five concise bullets.

There are three especially important arguments here.

CUDA_VISIBLE_DEVICES=0 exposes only the first GPU to the process. Again, we are making the one-GPU constraint explicit.

-ngl 99 asks llama.cpp to place enough model layers on the GPU that the entire model can be offloaded when memory allows. Supplying a number above the model’s actual layer count is a common way of saying “put all available layers on the accelerator.”

-c 8192 keeps the working context at a sensible local size rather than trying to exploit the model’s maximum context immediately.

The official llama.cpp project continues to support low-bit models and GPU offload as core features, including 4-bit integer formats and CUDA execution.

Why the two 4-bit approaches are not interchangeable

Both examples are called “4-bit,” but they are different systems.

With bitsandbytes:

You start from the original Transformers checkpoint.
Quantization is configured in Python.
The resulting model plugs naturally into the Transformers ecosystem.
Your application can manipulate processors, generation parameters, tensors, logits, and model internals directly.

With llama.cpp:

You normally start with a pre-converted low-bit file.
Quantization decisions were made when that file was produced.
The runtime is purpose-built for efficient local execution.
You can run without constructing a Python ML application around the model.
CPU/GPU hybrid execution remains available if the full model cannot reside on the accelerator.

Neither path is categorically “better.”

For experimentation, fine-tuning-adjacent workflows, or an existing Python stack, bitsandbytes is usually the more natural interface.

For a self-contained local assistant, desktop integration, command-line workflow, or compact server process, llama.cpp is often operationally simpler.

The best comparison is therefore empirical: run the same prompts through both and measure memory, latency, throughput, and output quality.

A useful quality test is not “does it produce English?”

A heavily damaged model can still produce fluent prose.

That makes casual inspection a poor quantization benchmark.

If you intend to use the model for code review, evaluate code review. If you want structured extraction, test exact extraction accuracy. If you want an assistant to follow formatting constraints, count formatting failures.

A small local evaluation suite can be surprisingly effective.

For example, create this neutral prompt set:

[
  {
    "id": "database_index",
    "prompt": "Explain why a database index can speed up reads but slow down writes. Use exactly five bullets."
  },
  {
    "id": "python_bug",
    "prompt": "Find the bug in this Python expression and provide a corrected version: values = [1, 2, 3]; print(values[3])"
  },
  {
    "id": "json_constraint",
    "prompt": "Return valid JSON only with keys city, temperature_c, and raining for this statement: Berlin is 19 C and dry."
  },
  {
    "id": "sorting",
    "prompt": "Sort these product codes lexicographically and output only the sorted list: Z9, A12, A2, B01."
  },
  {
    "id": "reasoning",
    "prompt": "A warehouse packs 24 items per box. How many full boxes and leftover items result from 1,003 items? Give only the two numbers."
  }
]

The point is not that these five prompts form a serious benchmark.

The point is that the outputs are checkable.

Run exactly the same set against:

The unquantized reference, if you have hardware that can run it.
The bitsandbytes 4-bit version.
The Q4 file.
Any smaller model you are considering as an alternative.

Record failures, not vibes.

A quantized model that uses less memory but breaks your JSON schema 15% more often may be a poor trade for an automation system. The same degradation might be irrelevant for informal brainstorming.

The cherry on the cake: smaller precision can beat the smaller model

One of the most interesting quantization results from the past year challenges a common intuition.

You might assume that the safest hierarchy is always:

larger FP16 model > larger quantized model > smaller FP16 model

A 2025 systematic study of language-model quantization found cases where that ordering did not hold.

In its code-generation experiments, the researchers reported that several quantized 34B configurations outperformed a 13B FP16 model in output quality while also achieving lower latency without additional energy cost in the reported comparison. More broadly, the paper found that a larger quantized model could sometimes provide a better quality/performance trade-off than a smaller full-precision model.

That is the surprising result worth remembering:

When memory is fixed, spending your bits on more parameters can sometimes be better than spending them on higher precision.

But there is an equally important warning hidden in the same study.

Quantization quality was not monotonic with the number of bits. Some configurations produced severe quality loss, and the paper reported HumanEval degradation as high as 92% in one tested setting. Simply saying “8-bit should be safer than 4-bit” is therefore not a sufficient engineering argument; the specific quantization method matters.

Do not copy those percentages onto our Qwen3.5 experiment. The study evaluated open Llama-family models with TensorRT-LLM on A100 and H100 hardware, not Qwen3.5 on a desktop card. The authors explicitly identify those platform and model choices as limitations.

The transferable lesson is methodological:

Benchmark the actual model, quantizer, runtime, hardware, and workload you intend to deploy.

“4-bit” by itself predicts surprisingly little about end-to-end quality.

What to do when 4-bit still runs out of memory

An out-of-memory error does not necessarily mean you need a smaller model.

Work through the budget systematically.

Reduce context first

If the model loads successfully but crashes while processing a long prompt, context is the obvious first lever.

With llama.cpp, retry at 4K:

CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-cli \
  -hf lmstudio-community/Qwen3.5-9B-GGUF:Q4_K_M \
  -ngl 99 \
  -c 4096

For the Transformers script, shorten the input and keep max_new_tokens conservative while establishing a baseline.

Check for other GPU processes

Run:

nvidia-smi

A few gigabytes held by another process can be the difference between a stable model and an apparently mysterious crash.

Do not confuse CPU offload with success

llama.cpp supports CPU/GPU hybrid execution, and Transformers can also place modules across devices in suitable configurations. Those capabilities are useful, but they answer a different question.

If your goal is one-GPU execution, verify that the model really resides there.

Try another quantization level

The .gguf ecosystem commonly offers several quantizations of the same model. A smaller file may recover enough headroom for your desired context, while a larger quantization may improve quality if you have spare VRAM.

Do not choose solely from the file size.

Run your evaluation prompts against the candidates.

What not to optimize too early

Once local inference works, it is easy to disappear into a maze of flags.

Resist that temptation.

First establish four baseline measurements:

Peak GPU memory.
Time to first useful output.
Sustained generation speed.
Accuracy on a small task-specific evaluation set.

Only then experiment with context sizes, quantization variants, sampling parameters, batching, speculative decoding, cache choices, or alternative runtimes.

Otherwise you can spend hours making tokens arrive faster while quietly reducing the percentage of answers you can trust.

A practical decision framework

If you already have a Python application and want the fastest route to a quantized Hugging Face model, start with bitsandbytes.

You retain the normal Transformers model interface, can change quantization settings in code, and avoid maintaining a separate conversion pipeline. Current bitsandbytes integrations expose 4-bit loading, NF4, configurable compute precision, and nested quantization directly through BitsAndBytesConfig.

If your priority is a compact standalone runtime, start with llama.cpp.

Its current tooling is explicitly designed around efficient local model execution, low-bit formats, accelerator backends, and optional CPU/GPU hybrid operation.

If you are deciding between a smaller model in high precision and a larger model in 4-bit precision, do not assume the smaller full-precision model wins.

The recent benchmark result above is a strong reason to test both.

And if a model technically fits but leaves only a few hundred megabytes of VRAM free, treat that as a warning rather than a victory. You still need room for the context, generation buffers, and the rest of your application.

The mental model to keep

The most useful way to think about local model deployment is as a budget.

Your GPU memory must pay for:

weights + quantization metadata + runtime state + context-dependent state + temporary buffers + framework overhead

Quantization dramatically shrinks the largest fixed component: the weights.

It does not repeal the rest of the bill.

A good 4-bit deployment therefore has three properties:

It fits with headroom. The model survives realistic prompts instead of merely completing initialization.
It is measured. You know the real peak VRAM and generation speed on your card.
It is evaluated. You have evidence that the low-bit model still performs your actual task correctly.

That third property is what separates a demo from an engineering decision.

Your next step

Pick one of the two runnable paths above and run Qwen3.5-9B on your own GPU today.

Do not stop when you see the first generated sentence. Record the peak VRAM, generation speed, and results from the five checkable prompts. Then try a second quantization and compare the numbers.

The goal is not simply to prove that a capable model can fit on one consumer GPU.

The goal is to find the smallest representation that still does your work correctly — and to have measurements that prove it.