Quantizing a vibration-anomaly model for a $60 edge board

INT8 on a Cortex-M55 without losing the tail of the distribution — and what the quantization-aware retrain actually bought us.

Condition monitoring lives in the tail. The anomalies you care about are, by definition, rare — and they are exactly the inputs a naive INT8 quantization is most likely to mangle. This is a write-up of getting a small vibration-anomaly model onto a ~$60 Cortex-M55 board without throwing away the tail of the score distribution.

The model

A compact 1D-CNN over accelerometer windows, trained on synthetic bearing-fault signatures plus public vibration datasets. Float32 baseline AUROC was strong; the question was how much survived quantization.

Post-training quantization, first pass

Straight PTQ with a small calibration set looked fine on aggregate accuracy and was quietly broken where it mattered. The score histogram showed the healthy class essentially intact, but the anomalous tail compressed toward the decision boundary — precisely the failure you cannot see if you only watch top-line accuracy.

What fixed it

Three changes, in order of impact:

Calibration set design. Calibrating on a representative sample that included near-threshold anomalies, not just easy healthy windows, recovered most of the separation.
Per-channel quantization for the convolutional weights instead of per-tensor.
A short quantization-aware retrain. A few epochs with fake-quant in the loop pulled the tail back out and bought the last bit of margin.

The numbers that matter

Model fit comfortably in the board's SRAM budget with headroom.
Per-window inference left plenty of duty-cycle for the sampling loop.
The anomalous-tail separation, not aggregate accuracy, was the metric that drove every decision.

Takeaway

For anomaly work, watch the tail of the score distribution, not the accuracy number. Quantization can hold aggregate metrics flat while quietly destroying the thing you deployed the model to catch.