Knowledge Base / int8-tiny-targets

Edge deploymentResearch

Does INT8 quantization hurt small object detection? The damage mAP does not show

Quantizing a detector to INT8 costs small objects 1.4 to 3 times more than the overall metric, and below 12 pixels the published loss reaches a third of the accuracy; what is known, the fixes, and what nobody has measured yet.

One quantization step can swallow a tiny target

One cross-section through an activation map. Lower the tiny target's contrast: below about half a step it rounds into the background and the INT8 curve goes flat. Raise the outlier: it sets the scale, so every step gets coarser. Switch to the percentile scale to clip the outlier and get the fine steps back. The recall over all targets barely moves, because tiny targets are 5 of 100.

012345activation↑ 12.0largetiny↑ 127.0outlier
0.30 (0.30 INT8 steps)
127
INT8 scale

INT8 step: 1.000 (scale = 127.0 / 127)

Tiny target after INT8: lost. In FP32: detected.

Recall over a fixed population of 100 targets (95 large, 5 tiny)
FP32INT8
All 100100% (100/100)95% (95/100)
Tiny 5100% (5/5)0% (0/5)

A simplified illustration of the mechanism, not a measurement: one cross-section, a made-up population and a toy detection rule. Activations are in units where 1 is one INT8 step at the default outlier (127).

Detection rule, the same for FP32 and INT8: a target counts as detected when its peak rises more than 0.1 above the local background (the median of 6 positions on each side). After INT8 the rise is zero or a whole number of steps, so the target survives only if its peak rounds up by at least one step, which on a background that rounds to zero means standing more than half a step tall.

Population: 95 large targets (peaks 1.5 to 18) and 5 tiny ones (peaks 0.8 to 1.2 times the slider value), each on its own dim background, all drawn from one fixed seed. One scale, calibrated on all of them together, applies to every target, as in static post-training quantization.

The problem

To run a detector on a small computer next to the camera, you compress it: weights and activations go from 32-bit floats to 8-bit integers (INT8), which made a small detector about twice as fast on a Raspberry Pi 5 CPU in one community measurement. The usual check is the overall accuracy (mAP) before and after, and it usually moves by a point or two. But the overall number averages over every object in the test set, and most of them are large. When the target you care about is a few pixels wide (a drone at a kilometre in a thermal camera, a bird, a person in an aerial image), the same compression can cost it far more, and the overall metric hides it. This article collects what is published about that effect, the fixes that exist, and what is still unmeasured.

The usual approach

  • Train in FP32, then apply post-training quantization (PTQ): run a few hundred calibration images, record the range of every activation tensor, and map that range onto 256 integer levels.
  • One scale per tensor for activations (per channel for weights), chosen by the maximum, a percentile, or the KL divergence between the float and integer histograms.
  • Accept the model if overall mAP drops by less than about one point; switch to quantization-aware training (QAT) if it drops more.
  • Report accuracy on COCO-style averages. COCO’s “small” bin is everything under 32 x 32 pixels, which is large compared with a 4-pixel target.

Where it fails, with numbers

Small objects lose more, consistently. Relative drops, computed from each paper’s own tables (AP is the overall metric, AP_S the small-object one, AP_L the large-object one):

paper setting AP AP_S AP_L
FQN, CVPR 2019 RetinaNet-R50, 4-bit QAT -8.7 % -14.8 % -8.4 %
FQN, CVPR 2019 Faster R-CNN-R50, 4-bit -12.2 % -17.3 % -10.8 %
AQD, CVPR 2021 FCOS-R50, 4-bit -2.3 % -7.2 % -2.2 %
Q-DETR, CVPR 2023 DETR-R50, 4-bit (attention 8-bit) -6.2 % -13.7 % -2.0 %
Q-DETR, a baseline in its tables SMCA-DETR, 8-bit PTQ, percentile calibration -8.5 % -19.6 % -5.4 %
Q-YOLO YOLOv5s / v7, 8-bit weights and activations, the Q-YOLO method 0 to -0.2 % -0.3 to -0.9 % -0.2 %

The pattern: under moderate damage, small objects lose 1.4 to 3 times more than the overall number and 1.6 to 7 times more than large ones; in the CNN rows the overall number sits close to the large-object one, because large objects dominate the count. The last row matters too: for a CNN YOLO, a careful 8-bit method costs COCO-small objects under one percent. Transformer detectors lose more: the best 8-bit method in Q-DETR’s tables (VT-PTQ) still costs small objects 7.3 %. At COCO scale and with a good recipe, INT8 is close to safe for a CNN.

Below 16 pixels it is no longer small. Two measurements, neither peer-reviewed, point the same way:

  • A bachelor thesis at Leiden (LIACS, 2026, PDF) quantized YOLOv11n to INT8 with ONNX Runtime PTQ (min-max calibration) on XS-VID, a dataset whose mean object is 10.6 pixels, with a ByteTrack tracker on top. Overall AP fell 10.5 %, but 35 % for objects of 0 to 12 pixels, 22 % for 12 to 20 and 15 % for 20 to 32. At track level, MOTA went from 0.247 to 0.235 and IDF1 from 0.404 to 0.376. Its QAT came out worse than PTQ and its proposed fix did not work, so read it as an indication, not a proof. It is the only track-level data we found on a dataset whose mean object is about 11 pixels (the tracking numbers are for the whole dataset, not per size).
  • A community measurement (hailo-uav-tracker, DUT Anti-UAV, YOLOv8n, ONNX Runtime INT8 as a stand-in for a hardware quantizer): FP32 to INT8 cost 6.8 % of overall AP, 14.4 % of small-object AP and 1.1 % of large-object AP. On a Raspberry Pi 5 CPU the INT8 model ran in 75.0 ms against 153.6 ms.

The mechanism. One activation scale per tensor is set by its largest values. In a thermal or aerial scene those are background outliers: sun glint, a hot roof, a cloud edge. The faint response of a tiny target in the high-resolution feature maps (the P2 and P3 levels of a feature pyramid) then spans one or two quantization steps, and rounding merges it into the background. A large target spans many steps and survives. A 2026 ISPRS paper argues, assuming the error averages over an object’s pixels, that the signal-to-quantization-noise ratio per object falls as its pixel count falls. The figure on this page is a simplified illustration of this mechanism, not a measurement.

Known fixes

fix what it does where it was tested limits
Better calibration (percentile, KL) and per-channel scales stops one outlier from setting the step for the whole tensor standard in TensorRT, ONNX Runtime, ncnn helps at COCO scale (Q-YOLO above); not shown to be enough below 12 px
OSA regularization, ISPRS Annals 2026 8-bit QAT with a feature-alignment loss on P2 / P3, weighted by 1 / object area YOLOX-Nano and Tiny, small birds and aerial buildings, on a Kria KV260 FPGA needs training, not calibration; reports overall AP only, no size bins
MLFQ, IET Image Processing 2026 (Wiley) PTQ block reconstruction with a spatial weight map from object sizes VisDrone, UAVDT we read the abstract only: gains at 4 bits and below, mixed at 8; numbers left out
LiDAR-PTQ, ICLR 2024 a foreground-weighted PTQ loss 3D detection another modality
Mixed precision keeps the detection head or the feature pyramid at higher precision various detectors not aimed at tiny objects on a CPU
ScaleKD, CVPR 2023 scale-aware distillation into a small detector COCO, VisDrone about model size, not quantization
CSCWD, 2026 distils a P2-head YOLO11m into YOLO11n for anti-UAV, runs it on a Pi 5 through ncnn Drone-vs-Bird; DUT Anti-UAV zero-shot FP16 only, no INT8, no size study

A related lever sits before the network. Thermal cameras deliver 14 or 16 bits per pixel, and the usual automatic gain control squeezes that into 8 bits before the detector sees it, which is a quantization step of its own. Task-adaptive tone mapping for thermal images exists (Thermal Chameleon, RA-L 2024), and so does adaptive 14-to-8-bit rescaling (Fieldscale, RA-L 2024). Nobody we found has studied it together with INT8 inference.

What we measured

Nothing yet. A measurement on a Raspberry Pi 5 is planned, and its rules are fixed before it runs:

  • Data: the thermal anti-UAV benchmark (Anti-UAV410 and CST Anti-UAV), single-object tracking, where a target with a box diagonal of 10 pixels or less is “tiny”.
  • What is reported: per target-size bin and per contrast bin, recall at a matched false-alarm rate, the benchmark’s state accuracy (SA), lock losses per minute and time to first detection; per layer, the target-to-background activation ratio in FP32 and INT8; latency for every configuration, with the device card.
  • Hypothesis: at 8 bits, recall on targets of 10 pixels or less falls at least twice as much as overall AP, and the loss shows up at track level as more lock losses and a later first detection.
  • What would prove it wrong: tiny-target recall falls no more than overall AP. Then the effect is a COCO-scale artefact, not a property of this setting, and this article will say so.

The result, either way, will be added here with its numbers.

Open questions

  1. Thermal, 3 to 16 pixels. No study of quantization damage on thermal anti-UAV data at these sizes.
  2. Single-object tracking. Lock loss, re-acquisition time and time to first detection under INT8 are unmeasured; the only track-level study is RGB and multi-object.
  3. ARM CPU. Prior work runs on GPUs (TensorRT) and FPGAs; the CPU path through ncnn, the common one on a Raspberry Pi, is unstudied.
  4. Contrast, not only size. Results are binned by size; in thermal data a 6-pixel target can be easy or invisible. Is contrast the better predictor?
  5. Raw bit depth. How does 14 or 16-bit input interact with an INT8 first layer, which quantizes the input anyway?
  6. A recipe at 8 bits. Is there a cheap way to recover tiny-target recall at 8 bits, at near-INT8 latency? Published size-aware fixes either need training or pay off only at 4 bits and below.
  7. Which layers to protect. No paper keeps exactly the early, high-resolution layers at FP16 for tiny targets on a CPU, and no rule says which layers to protect from target size and layer stride.

Sources