Optimizing YOLOv8 Object Inference to Sub-10ms Using TensorRT and Docker

A PyTorch YOLOv8 checkpoint running through ultralytics inference in eager mode is a reasonable starting point, but it leaves real latency on the table once you have a fixed deployment target. Getting from "works in a notebook" to "sub-10ms per frame in production" is mostly about narrowing what the runtime has to guess at.

Step 1: Export to ONNX, Then Compile with TensorRT

yolo export model=best.pt format=onnx opset=17 dynamic=False
trtexec --onnx=best.onnx --saveEngine=best.engine --fp16

Fixing dynamic=False matters — a static input shape lets TensorRT fuse layers and pick kernels for that exact shape instead of keeping the flexibility (and overhead) needed for arbitrary input sizes.

Step 2: INT8 Calibration, Not Just FP16

FP16 gets you roughly 2x over FP32 with almost no accuracy loss. INT8 gets you further, but needs a calibration pass over representative data so TensorRT can pick sane quantization ranges per layer:

trtexec --onnx=best.onnx --saveEngine=best_int8.engine \
  --int8 --calib=calibration_cache.bin

Skipping calibration and quantizing blind is the most common way to quietly tank mAP — always validate accuracy on a held-out set after this step, not just latency.

Step 3: Container Layout That Avoids Cold Starts

The engine file is hardware-specific — a TensorRT engine built on one GPU architecture won't load on another. Bake the compiled engine into the image at build time, not at container start:

FROM nvcr.io/nvidia/tensorrt:24.01-py3
COPY best_int8.engine /models/
COPY server.py /app/
CMD ["python", "/app/server.py"]

This turns "compile TensorRT engine" from a per-request or per-boot cost into a one-time build step, which is most of the win once the model itself is already fast.

Result

Combining static-shape ONNX export, INT8 calibration, and a prebuilt-engine container gets a YOLOv8n detection pass from roughly 25-30ms (PyTorch eager, FP32) down under 10ms per frame on comparable hardware — the difference between "usable for a dashboard" and "usable for a live multi-stream tracking pipeline" like the one behind GuardianNet AI.

Numbers above are representative of this optimization path in general; exact figures depend on the specific GPU and batch size used.