Embeddings as features: classify text with sentence embeddings and logistic regression

LEARN · FEATURE ENGINEERING & THE SCIKIT-LEARN TOOLKIT

Why this pattern is worth learning

Modern text classification does not always need a model fine-tuned end to end.

A surprisingly strong architecture is much simpler:

Run each text through a pretrained sentence-embedding model.
Freeze those embeddings as ordinary numeric features.
Train a conventional classifier such as logistic regression on top.
At inference time, embed new text and feed the resulting vector into the classifier.

The pipeline is:

text
  ↓
pretrained embedding model
  ↓
dense semantic vector
  ↓
logistic regression
  ↓
class probabilities

This gives you an interesting division of labor. The embedding model does the expensive semantic representation work. Logistic regression learns where your application-specific class boundaries sit in that embedding space.

For many practical classification problems, especially when labeled data is scarce, this is an excellent baseline before considering transformer fine-tuning.

It is particularly useful when your classes describe meanings rather than specific keywords:

customer-support intents;
routing incoming messages;
product-feedback categories;
moderation queues;
document types;
issue-tracker labels;
e-commerce request classification;
internal knowledge-base taxonomy.

The central idea is that a sentence embedding can place two lexically different sentences close together because they express similar ideas. A bag-of-words model sees vocabulary. A strong embedding model can see something closer to intent.

That difference becomes important as soon as production users stop phrasing things exactly like your training examples.

The model stack we will use

This tutorial uses the current Sentence Transformers API with Qwen/Qwen3-Embedding-0.6B.

Qwen3-Embedding-0.6B is a 2025 text-embedding model with approximately 0.6 billion parameters, support for more than 100 languages, a context length of up to 32K tokens, and embeddings up to 1024 dimensions. It supports Sentence Transformers directly and also supports Matryoshka-style reduced embedding dimensions.

For the software stack, Sentence Transformers 6.0.1 was released on August 31, 2026, while scikit-learn 1.9.0 was released on June 2, 2026. Sentence Transformers currently requires Python 3.10 or newer, while scikit-learn 1.9.0 requires Python 3.11 or newer, so Python 3.11+ is the practical shared minimum for this project.

Create an environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install "sentence-transformers==6.0.1" "scikit-learn==1.9.0"

The first execution will download the embedding model. Qwen3-Embedding-0.6B’s primary model weights are roughly 1.2 GB, so this is not a tiny download, although inference remains far lighter than serving a multi-billion-parameter generative model.

What an embedding becomes from scikit-learn’s perspective

Suppose the model converts this message:

Could you reroute my parcel to my new apartment?

into a 1024-dimensional vector.

Conceptually, it looks like this:

[0.018, -0.041, 0.007, ..., 0.029]

Scikit-learn does not care that those values came from a transformer. As far as logistic regression is concerned, each embedding dimension is simply a numeric feature.

If you have 500 labeled messages, the resulting feature matrix might have shape:

(500, 1024)

Logistic regression then learns one or more linear decision boundaries in that semantic space.

For a multiclass problem, you can think of the classifier as computing a score for every class from the embedding and converting those scores into probabilities.

The key word is semantic.

A traditional word-level TF-IDF feature space might strongly distinguish:

package
parcel
shipment
delivery

because they are different tokens.

A good sentence embedding attempts to represent the meaning shared by sentences containing those words.

That means your classifier can sometimes generalize to vocabulary that never appeared in the labeled training data.

Why normalize embeddings?

Sentence Transformers supports normalization directly through the current encode() API using normalize_embeddings=True.

We will use it throughout this tutorial.

After L2 normalization, every embedding has length 1. This gives two useful properties:

feature magnitudes are kept on a consistent scale;
the dot product between two normalized embeddings equals their cosine similarity.

The second property will become especially useful for our nearest-neighbor experiment later.

For classification, normalization is not mandatory. You should validate it for your own model and task. But when embeddings were trained around cosine-style semantic geometry, normalized vectors are often a sensible starting point.

Build a benchmark that actually tests semantic generalization

A weak benchmark can make almost any model look good.

Imagine the training set contains:

Where is my package?

and the test set contains:

Where is my package now?

A TF-IDF classifier will probably perform well because the lexical overlap is enormous. That does not tell us whether embeddings are buying us much.

Instead, we will deliberately create a lexical-shift split.

Training examples and test examples represent the same six e-commerce support intents, but the test set often uses different wording.

For example:

Training: Please cancel my order before it ships.
Test: Abort the transaction before fulfillment begins.

A semantic model should understand that these requests are related even though the wording changes substantially.

That is exactly the kind of generalization we want to test.

The complete experiment

Create a file named train.py:

from pathlib import Path

import joblib
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.pipeline import Pipeline


EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-0.6B"
ARTIFACT_PATH = Path("intent_classifier.joblib")


TRAIN_DATA = [
    ("Where is my package?", "delivery_status"),
    ("Can I track my shipment?", "delivery_status"),
    ("When will the delivery arrive?", "delivery_status"),
    ("The courier has not shown up yet.", "delivery_status"),
    ("My package seems delayed.", "delivery_status"),
    ("Please tell me the shipment location.", "delivery_status"),
    ("Is my delivery still on the way?", "delivery_status"),
    ("I need the tracking progress for my package.", "delivery_status"),

    ("I want to return this item.", "return_request"),
    ("How do I send my purchase back?", "return_request"),
    ("Please start a refund for this item.", "return_request"),
    ("I no longer want this product.", "return_request"),
    ("Can I exchange what I bought?", "return_request"),
    ("Where do I send a return?", "return_request"),
    ("I need a refund for my purchase.", "return_request"),
    ("Please help me return the product.", "return_request"),

    ("Please cancel my order.", "cancel_order"),
    ("Stop the order before it ships.", "cancel_order"),
    ("I do not want this order anymore.", "cancel_order"),
    ("Can you cancel the purchase immediately?", "cancel_order"),
    ("Do not send the order.", "cancel_order"),
    ("I need to revoke my purchase.", "cancel_order"),
    ("Please stop processing my order.", "cancel_order"),
    ("Cancel this purchase before dispatch.", "cancel_order"),

    ("I entered the wrong shipping address.", "change_address"),
    ("Please change where my order is being shipped.", "change_address"),
    ("I need to update my delivery address.", "change_address"),
    ("Can you ship the package somewhere else?", "change_address"),
    ("The destination on my order is incorrect.", "change_address"),
    ("Please update the address before shipping.", "change_address"),
    ("I want my package sent to another location.", "change_address"),
    ("Can I modify the shipping destination?", "change_address"),

    ("My card was declined.", "payment_problem"),
    ("The payment failed during checkout.", "payment_problem"),
    ("Why was my card rejected?", "payment_problem"),
    ("I cannot complete the payment.", "payment_problem"),
    ("Checkout keeps failing when I pay.", "payment_problem"),
    ("My card charge will not go through.", "payment_problem"),
    ("There is an error when I try to pay.", "payment_problem"),
    ("The store refuses my payment method.", "payment_problem"),

    ("What material is this product made from?", "product_question"),
    ("What are the dimensions of this item?", "product_question"),
    ("Is this product compatible with my laptop?", "product_question"),
    ("Which sizes are available?", "product_question"),
    ("Is this item currently in stock?", "product_question"),
    ("Can you tell me the product measurements?", "product_question"),
    ("What colors does this item come in?", "product_question"),
    ("Will this accessory work with my device?", "product_question"),
]


TEST_DATA = [
    ("Has my parcel made any progress toward me?", "delivery_status"),
    ("What is the current whereabouts of the parcel?", "delivery_status"),
    ("When should the box reach my doorstep?", "delivery_status"),
    ("Has the carrier moved my parcel closer yet?", "delivery_status"),

    ("Can I give the merchandise back and get reimbursed?", "return_request"),
    ("I would like to reverse this purchase after receiving it.", "return_request"),
    ("How can I hand the unwanted merchandise back?", "return_request"),
    ("I need reimbursement for something I already received.", "return_request"),

    ("Abort the transaction before fulfillment begins.", "cancel_order"),
    ("I want to withdraw the purchase before anything is dispatched.", "cancel_order"),
    ("Please prevent fulfillment of what I just bought.", "cancel_order"),
    ("Can the transaction be called off before it leaves the warehouse?", "cancel_order"),

    ("I moved home and need the parcel redirected.", "change_address"),
    ("Can you reroute the box to my new apartment?", "change_address"),
    ("The drop-off location needs to be changed.", "change_address"),
    ("Please redirect the consignment to a different residence.", "change_address"),

    ("The transaction is being rejected by the bank.", "payment_problem"),
    ("I cannot get the purchase authorized.", "payment_problem"),
    ("My bank will not approve the transaction.", "payment_problem"),
    ("The transaction fails every time I attempt to complete it.", "payment_problem"),

    ("What fabric is used to make this?", "product_question"),
    ("How wide and tall is the item?", "product_question"),
    ("Will this fit the computer I own?", "product_question"),
    ("Which variants can I currently buy?", "product_question"),
]


def unpack(rows):
    texts = [text for text, _ in rows]
    labels = [label for _, label in rows]
    return texts, labels


train_texts, train_labels = unpack(TRAIN_DATA)
test_texts, test_labels = unpack(TEST_DATA)


print("Training TF-IDF baseline...")

tfidf_classifier = Pipeline(
    [
        (
            "tfidf",
            TfidfVectorizer(
                lowercase=True,
                stop_words="english",
                ngram_range=(1, 2),
                sublinear_tf=True,
            ),
        ),
        (
            "classifier",
            LogisticRegression(
                penalty="l2",
                solver="lbfgs",
                C=4.0,
                max_iter=3000,
            ),
        ),
    ]
)

tfidf_classifier.fit(train_texts, train_labels)
tfidf_predictions = tfidf_classifier.predict(test_texts)
tfidf_accuracy = accuracy_score(test_labels, tfidf_predictions)

print()
print(f"TF-IDF accuracy: {tfidf_accuracy:.3f}")
print(classification_report(test_labels, tfidf_predictions, zero_division=0))


print("Loading sentence embedding model...")

encoder = SentenceTransformer(EMBEDDING_MODEL)

train_embeddings = encoder.encode(
    train_texts,
    batch_size=16,
    show_progress_bar=True,
    normalize_embeddings=True,
)

test_embeddings = encoder.encode(
    test_texts,
    batch_size=16,
    show_progress_bar=True,
    normalize_embeddings=True,
)

print()
print(f"Training embedding matrix: {train_embeddings.shape}")
print(f"Test embedding matrix: {test_embeddings.shape}")


embedding_classifier = LogisticRegression(
    penalty="l2",
    solver="lbfgs",
    C=4.0,
    max_iter=3000,
)

embedding_classifier.fit(train_embeddings, train_labels)
embedding_predictions = embedding_classifier.predict(test_embeddings)
embedding_accuracy = accuracy_score(test_labels, embedding_predictions)

print()
print(f"Embedding + logistic regression accuracy: {embedding_accuracy:.3f}")
print(
    classification_report(
        test_labels,
        embedding_predictions,
        zero_division=0,
    )
)


print("Comparing the two feature representations...")

delta = embedding_accuracy - tfidf_accuracy
print(f"Accuracy improvement over TF-IDF: {delta:+.3f}")


print()
print("Nearest-neighbor experiment...")

similarities = test_embeddings @ train_embeddings.T
nearest_indices = np.argmax(similarities, axis=1)
nearest_predictions = np.asarray(train_labels)[nearest_indices]
nearest_scores = similarities[
    np.arange(len(test_embeddings)),
    nearest_indices,
]

nearest_accuracy = accuracy_score(test_labels, nearest_predictions)
agreement = np.mean(nearest_predictions == embedding_predictions)

print(f"1-NN embedding accuracy: {nearest_accuracy:.3f}")
print(f"1-NN / logistic-regression agreement: {agreement:.3f}")


print()
print("Nearest examples:")

for text, true_label, lr_label, nn_label, nn_index, score in zip(
    test_texts,
    test_labels,
    embedding_predictions,
    nearest_predictions,
    nearest_indices,
    nearest_scores,
):
    nearest_text = train_texts[nn_index]

    print()
    print(f"Text: {text}")
    print(f"True label: {true_label}")
    print(f"Logistic regression: {lr_label}")
    print(f"Nearest-neighbor label: {nn_label}")
    print(f"Nearest training example: {nearest_text}")
    print(f"Cosine similarity: {score:.3f}")


joblib.dump(
    {
        "embedding_model": EMBEDDING_MODEL,
        "classifier": embedding_classifier,
    },
    ARTIFACT_PATH,
)

print()
print(f"Saved classifier to {ARTIFACT_PATH}")

Run it:

python train.py

Do not focus only on the final accuracy number.

Study three outputs:

TF-IDF accuracy;
embedding-plus-logistic-regression accuracy;
nearest-neighbor accuracy and agreement.

On this dataset, the test split has intentionally been written to reduce easy word overlap. That gives the embedding model an opportunity to demonstrate semantic transfer instead of merely memorizing recurring vocabulary.

If your embedding classifier fails to beat TF-IDF, do not hide the result. Inspect the errors. Classification experiments should measure reality rather than encode the expected result into an assertion.

Why TF-IDF struggles under lexical shift

TF-IDF is not a bad technique. In fact, for many text problems it remains extremely competitive.

Its limitation is representation.

Consider:

Training: My card was declined.
Test: The transaction is being rejected by the bank.

A word-level TF-IDF model cannot infer from first principles that card declined and transaction rejected by the bank are related concepts.

If it has previously observed enough overlapping vocabulary across many examples, it can learn useful statistical associations. But with a small dataset and substantial vocabulary shift, the representation itself becomes the bottleneck.

Sentence embeddings approach the problem differently.

The pretrained encoder has already learned relationships among phrases from large-scale representation training. Your tiny labeled dataset therefore does not need to teach the classifier everything about natural language.

It only needs to teach the linear model something closer to:

this region of semantic space means payment_problem

and:

that region means change_address

This is a major reason frozen embeddings can work well with surprisingly little labeled data.

Why logistic regression is such a good companion

It is tempting to assume that once a transformer is involved, the classifier on top also needs to be complicated.

Usually it does not.

Logistic regression has several advantages here:

training is extremely fast compared with transformer fine-tuning;
inference after embedding is cheap;
it produces class probabilities;
L2 regularization helps when the embedding dimension is large relative to the labeled dataset;
the model is easy to save, inspect, retrain, and monitor.

Scikit-learn’s current logistic-regression implementation accepts dense matrices directly and applies regularization by default. Its standard solvers also support native multiclass optimization where appropriate.

The transformer has already created nonlinear semantic features. The final decision boundary often does not need another deep neural network.

This is the same general pattern that makes pretrained representations powerful elsewhere: invest computation in learning useful features once, then solve downstream tasks cheaply.

Do not use retrieval-specific query/document encoding by accident

Sentence Transformers currently offers specialized encode_query() and encode_document() methods for retrieval applications. Those methods can apply different prompts and route inputs differently when a model supports retrieval-specific behavior.

That is useful when your task is asymmetric:

search query → retrieve document

Classification is normally symmetric.

Your training messages and production messages represent the same kind of object, so encode them consistently.

That is why the tutorial uses:

embeddings = encoder.encode(
    texts,
    normalize_embeddings=True,
)

for both training and inference.

Do not encode training messages as documents and incoming messages as queries merely because those APIs exist. Doing so could put the two sets through different task-specific behavior.

Predict new messages

The training script stored the logistic-regression classifier plus the embedding model identifier.

Create predict.py:

import joblib
from sentence_transformers import SentenceTransformer


bundle = joblib.load("intent_classifier.joblib")

encoder = SentenceTransformer(bundle["embedding_model"])
classifier = bundle["classifier"]

messages = [
    "Can you send the parcel to my office instead?",
    "The bank keeps refusing the transaction.",
    "How heavy is this product?",
]

embeddings = encoder.encode(
    messages,
    batch_size=16,
    normalize_embeddings=True,
)

probabilities = classifier.predict_proba(embeddings)
predictions = classifier.predict(embeddings)

for message, prediction, probability_row in zip(
    messages,
    predictions,
    probabilities,
):
    confidence = probability_row.max()

    print(f"Message: {message}")
    print(f"Prediction: {prediction}")
    print(f"Classifier probability: {confidence:.3f}")
    print()

Run it:

python predict.py

Only load serialized joblib or pickle-style artifacts that you trust. Python object deserialization should not be treated as a safe interchange format for untrusted files.

Add an abstention threshold

In production, a classifier should not necessarily be forced to classify everything.

Suppose your six known categories suddenly receive:

Please delete my account and all associated data.

None of the training classes describes that request.

Logistic regression still has to assign probability mass somewhere. A high maximum probability is evidence about the classifier’s decision, but it is not proof that the input belongs to one of your known categories.

You can introduce an abstention rule:

import joblib
from sentence_transformers import SentenceTransformer


THRESHOLD = 0.65

bundle = joblib.load("intent_classifier.joblib")
encoder = SentenceTransformer(bundle["embedding_model"])
classifier = bundle["classifier"]

messages = [
    "Please reroute my parcel.",
    "Delete every piece of personal data you have about me.",
]

embeddings = encoder.encode(
    messages,
    normalize_embeddings=True,
)

probabilities = classifier.predict_proba(embeddings)

for message, probability_row in zip(messages, probabilities):
    best_index = probability_row.argmax()
    best_probability = probability_row[best_index]
    best_label = classifier.classes_[best_index]

    if best_probability < THRESHOLD:
        prediction = "needs_review"
    else:
        prediction = best_label

    print(f"{prediction}: {message}")

The value 0.65 is only an example.

Do not choose an operational threshold by intuition. Create a held-out validation set containing both known-class examples and realistic out-of-scope messages, then select a threshold based on the cost of false acceptance versus manual review.

Also remember that raw logistic-regression probabilities are not automatically perfectly calibrated probabilities. If calibration matters operationally, evaluate calibration on held-out data rather than interpreting 0.90 as a guaranteed 90% real-world correctness rate.

The cherry on the cake: nearest-neighbor search may nearly replace the classifier

Look back at this line from the training script:

similarities = test_embeddings @ train_embeddings.T

Because we normalized both matrices, every dot product is a cosine similarity.

For each test message, we then find the training message with the highest semantic similarity:

nearest_indices = np.argmax(similarities, axis=1)
nearest_predictions = np.asarray(train_labels)[nearest_indices]

There is no trained classifier in that calculation.

It is simply semantic nearest-neighbor search.

And on well-separated intent datasets, something surprising can happen: the nearest-neighbor predictions can agree with the trained logistic-regression classifier on most or even all test examples.

The program measures the exact agreement:

agreement = np.mean(
    nearest_predictions == embedding_predictions
)

print(
    f"1-NN / logistic-regression agreement: {agreement:.3f}"
)

If that value reaches 1.000, your nearest-neighbor lookup has made exactly the same predictions as logistic regression on the entire test set.

That sounds strange until you think geometrically.

Suppose messages about payment failures naturally form one cluster:

card rejected
bank refuses transaction
payment authorization fails
checkout payment blocked

and address changes form another:

reroute my parcel
change destination
send it to my new apartment
redirect the package

Logistic regression learns a boundary between the clusters.

Nearest-neighbor search does something even simpler: it asks which known example is closest.

When embedding space already organizes your application concepts cleanly, both approaches can arrive at the same answer.

This has an important engineering implication.

Your embedding database is not useful only for classification. The same vectors can support:

classification;
semantic search;
similar-case retrieval;
example inspection;
duplicate detection;
clustering;
human-readable explanations.

For example, instead of merely telling an operator:

prediction = change_address

you can show:

Closest labeled example:
"I want my package sent to another location."

That nearest example often provides a far more intuitive debugging clue than examining 1024 logistic-regression coefficients.

Why nearest neighbor does not make logistic regression obsolete

The nearest-neighbor result is interesting, but do not conclude that classification training is pointless.

The two approaches encode different assumptions.

One-nearest-neighbor classification bases the decision on one labeled example. That can make it sensitive to:

mislabeled records;
unusual examples;
duplicates;
noisy boundary cases.

Logistic regression uses the full training set to estimate a global class boundary.

Imagine ten legitimate payment_problem examples clustered together but one incorrectly labeled training example sitting nearby. A nearest-neighbor classifier can copy the bad label. Logistic regression may be more resistant because its parameters are learned from the entire distribution.

The useful lesson is not “replace logistic regression with nearest neighbors.”

It is:

A strong embedding space can make multiple very simple downstream algorithms work surprisingly well.

That is evidence that representation quality often matters more than classifier complexity.

When embedding features can beat transformer fine-tuning

Fine-tuning has a higher performance ceiling when you have enough good labels and a task worth specializing for.

But it also introduces more moving parts:

GPU training;
learning-rate selection;
batch-size tuning;
checkpoint management;
early stopping;
catastrophic-overfitting risk on tiny datasets;
model versioning;
training reproducibility;
heavier deployment artifacts.

Frozen embeddings remove most of that.

This approach is especially attractive when you have small labeled datasets. If you have only tens or hundreds of labeled examples per class, adapting millions or billions of transformer parameters may provide less benefit than using a strong pretrained representation and learning a regularized linear boundary.

It also works well when labels change frequently. Suppose the business adds two new support categories every month. Re-embedding the old corpus may not even be necessary: if the embedding model remains unchanged, you can often add labeled examples and retrain only logistic regression.

That retraining can take seconds.

The method is also useful when one embedding model serves several tasks.

You might compute an embedding once and reuse it for:

message classification;
semantic retrieval;
duplicate detection;
clustering;
anomaly analysis.

Fine-tuning separate transformer checkpoints for every task removes much of that reuse.

Finally, frozen embeddings make experimentation cheap. You can test logistic regression, nearest centroids, linear SVMs, nearest neighbors, gradient boosting, or shallow neural networks without repeatedly running the transformer.

When fine-tuning should win

Do not turn the technique into dogma.

Fine-tuning becomes increasingly attractive when you have:

thousands or millions of reliable labels;
highly domain-specific vocabulary;
distinctions that generic semantic similarity does not capture;
enough compute for systematic experimentation;
a stable taxonomy;
evidence that the frozen representation is the main performance bottleneck.

Consider these two labels:

eligible_for_free_replacement
not_eligible_for_free_replacement

Messages in both categories may discuss almost identical products, defects, purchases, and replacement requests. The distinction could depend on tiny policy details such as purchase date, warranty status, or wording.

A generic embedding model may deliberately place those sentences close together because they are semantically similar.

Your classification task, however, wants them separated.

Fine-tuning can reshape the representation around that particular distinction.

That is the fundamental tradeoff.

Frozen embeddings ask:

Can a generic semantic space already separate my labels?

Fine-tuning asks:

Can I modify the semantic space so that it separates my labels better?

Always test the cheap option first.

A stronger experimental protocol

The tiny dataset above teaches the mechanics, not production evaluation.

For a real system, create three splits:

training;
validation;
final test.

Do not repeatedly tune against the final test set.

More importantly, make the split resemble deployment.

For text classification, random splitting can be dangerously optimistic when near-duplicate text exists.

Suppose your dataset contains:

Where is order 88341?
Where is order 11729?
Where is order 66201?

A random split may place two examples in training and one in testing. The model appears brilliant, but it has really seen nearly the same template.

Better splits may separate examples by:

customer;
conversation;
document;
product;
template;
time period;
source system.

A time-based holdout is particularly valuable when production language evolves.

The question is not:

Can the model recognize randomly hidden examples from yesterday’s dataset?

The question is:

Will the model classify the next batch of real messages correctly?

Cache embeddings instead of recomputing them

Embedding computation is usually the most expensive part of this architecture.

Logistic regression is cheap.

Once your training texts are embedded, save the resulting matrix.

For example:

import numpy as np
from sentence_transformers import SentenceTransformer


texts = [
    "Where is my parcel?",
    "Please cancel the purchase.",
    "My bank rejected the transaction.",
]

encoder = SentenceTransformer(
    "Qwen/Qwen3-Embedding-0.6B"
)

embeddings = encoder.encode(
    texts,
    normalize_embeddings=True,
)

np.save("train_embeddings.npy", embeddings)

Load them later:

import numpy as np


embeddings = np.load("train_embeddings.npy")

print(embeddings.shape)

This changes your experimentation loop dramatically.

Instead of:

new classifier idea
→ rerun transformer
→ train classifier

you get:

embed dataset once
→ try classifier A
→ try classifier B
→ tune regularization
→ inspect neighbors
→ cluster vectors
→ run error analysis

The expensive representation step is amortized across experiments.

Just remember that cached embeddings belong to a specific embedding model and preprocessing configuration. If you change the model, normalization behavior, truncation settings, or prompt strategy, regenerate the vectors.

Try smaller embedding dimensions carefully

Qwen3-Embedding-0.6B supports Matryoshka representation learning and output dimensions below its full 1024 dimensions. The model card documents supported dimensions from 32 through 1024.

Sentence Transformers exposes dimensional truncation through its embedding API.

For example:

from sentence_transformers import SentenceTransformer


encoder = SentenceTransformer(
    "Qwen/Qwen3-Embedding-0.6B"
)

texts = [
    "Where is my parcel?",
    "Please cancel this transaction.",
]

embeddings = encoder.encode(
    texts,
    normalize_embeddings=True,
    truncate_dim=256,
)

print(embeddings.shape)

The result should have 256 features per text instead of 1024.

That can reduce:

stored vector size;
memory bandwidth;
classifier input size;
similarity-search cost.

Do not assume 256 dimensions are automatically sufficient for your task. Benchmark 1024, 512, 256, and perhaps smaller representations against the same held-out test set.

The right dimension is an engineering tradeoff, not a universal constant.

Why we are not fine-tuning the embedding model

Notice what the training script never does.

It never calls an optimizer on Qwen’s parameters.

The encoder remains frozen.

Training consists only of:

embedding_classifier.fit(
    train_embeddings,
    train_labels,
)

This separation is powerful operationally.

You can version the system as two pieces:

embedding model:
Qwen/Qwen3-Embedding-0.6B

classification head:
intent_classifier.joblib

If labels change but language representation does not, retrain only the second piece.

If you later upgrade the embedding model, regenerate the embeddings and retrain the classifier.

This gives you a clean experimental boundary.

Never replace the embedding model underneath an existing logistic-regression classifier without retraining it. Different embedding models produce different coordinate systems, even when their output dimensions happen to match.

A classifier trained on one model’s vectors has no reason to work on another model’s vectors.

What to measure besides accuracy

Accuracy is convenient but incomplete.

For multiclass production systems, inspect:

per-class precision;
per-class recall;
F1 score;
confusion matrix;
class support;
abstention rate;
probability calibration;
latency;
embedding throughput.

Pay special attention to minority classes.

A classifier can achieve excellent overall accuracy while consistently failing on a rare but important category.

Also inspect examples where models disagree.

The most informative records are often:

TF-IDF correct, embedding classifier wrong
embedding classifier correct, TF-IDF wrong
logistic regression and nearest neighbor disagree
high confidence but wrong
low confidence but correct

These subsets reveal what each representation is actually learning.

Common mistakes

Training and inference use different embedding settings

If training uses normalized vectors:

train_embeddings = encoder.encode(
    train_texts,
    normalize_embeddings=True,
)

production should do the same:

new_embeddings = encoder.encode(
    new_texts,
    normalize_embeddings=True,
)

Changing preprocessing between training and inference creates feature drift before your application has even received real-world drift.

Re-embedding every time you tune logistic regression

Do not rerun the transformer just because you changed the regularization strength.

Cache the embeddings.

Assuming the highest probability means the label is valid

Closed-set classifiers always choose among known labels.

Add out-of-domain evaluation and an abstention path.

Evaluating on duplicates

Duplicate or templated messages can make every model appear stronger than it really is.

Group related records before splitting.

Skipping the cheap baseline

Always compare against TF-IDF.

Sometimes the supposedly primitive model wins.

If your classes are driven by explicit vocabulary such as product codes, error identifiers, technical tokens, or standardized phrases, sparse lexical features may be exactly what you need.

Assuming a newer embedding model automatically improves your classifier

New embedding benchmarks do not guarantee improvement on your specific taxonomy.

Whenever you swap embedding models:

regenerate embeddings;
retrain the classifier;
rerun the same held-out evaluation;
compare latency and memory as well as accuracy.

A practical decision ladder

For a new text-classification project, a sensible progression is:

Build a TF-IDF plus logistic-regression baseline.
Replace TF-IDF with frozen sentence embeddings.
Inspect nearest neighbors and classification errors.
Tune regularization and embedding dimensionality.
Add realistic validation and abstention logic.
Fine-tune only if frozen embeddings leave meaningful performance on the table.

This sequence minimizes engineering cost while continuously increasing model sophistication only when the data justifies it.

You may discover that step two is already good enough.

That is not a compromise. A frozen semantic encoder plus a tiny discriminative classifier can be a very strong production architecture.

The deeper lesson

The interesting part of this technique is not logistic regression itself.

It is the change in what your labeled dataset has to learn.

With TF-IDF, your training set must teach the classifier both vocabulary associations and class boundaries.

With pretrained sentence embeddings, much of the language understanding already lives inside the representation model. The labeled dataset mostly has to teach the downstream classifier how your own taxonomy maps onto that semantic space.

That is why a few dozen labeled examples can sometimes produce behavior that would have required far more data with lexical features.

It also explains the nearest-neighbor surprise.

If one normalized embedding of:

Can you reroute the box to my new apartment?

lands next to:

I want my package sent to another location.

then the representation model has already done most of the intellectually difficult work.

Logistic regression is simply drawing boundaries around that geometry.

Start with the supplied benchmark, then replace the synthetic support messages with a carefully split sample from your own non-sensitive domain. Compare TF-IDF, frozen embeddings, logistic regression, and nearest-neighbor search on exactly the same holdout set. If frozen sentence embeddings already separate your real classes cleanly, ship the simple classifier first—and make fine-tuning earn its additional complexity with measured improvements.

Why this pattern is worth learning

The model stack we will use

What an embedding becomes from scikit-learn’s perspective

Why normalize embeddings?

Build a benchmark that actually tests semantic generalization

The complete experiment

Why TF-IDF struggles under lexical shift

Why logistic regression is such a good companion

Do not use retrieval-specific query/document encoding by accident

Predict new messages

Add an abstention threshold

The cherry on the cake: nearest-neighbor search may nearly replace the classifier

Why nearest neighbor does not make logistic regression obsolete

When embedding features can beat transformer fine-tuning

When fine-tuning should win

A stronger experimental protocol

Cache embeddings instead of recomputing them

Try smaller embedding dimensions carefully

Why we are not fine-tuning the embedding model

What to measure besides accuracy

Common mistakes

Training and inference use different embedding settings

Re-embedding every time you tune logistic regression

Assuming the highest probability means the label is valid

Evaluating on duplicates

Skipping the cheap baseline

Assuming a newer embedding model automatically improves your classifier

A practical decision ladder

The deeper lesson

Embeddings as features: classify text with sentence embeddings and logistic regression

Vulnerability Watch No65

The lakehouse: Iceberg, Delta, and tables that time-travel