onnx-runtime-inference
verifieda620467d-5a4f-42f6-bfb1-370ac60bda63
Optimize local model inference with ONNX Runtime — execution providers, graph optimization, quantization, IO binding, and session reuse.
Metadata
Skill file
# Optimizing Local Inference with ONNX Runtime
Use when you've exported a model (PyTorch/TensorFlow) and want GPU/CPU inference
faster than the eager framework, or you need to cut latency for a local or
edge deployment — without rewriting your stack. ONNX Runtime (ORT) turns an
ONNX model into an optimized execution plan with pluggable execution providers.
## Pick and order execution providers
ORT tries providers in the order you pass them and falls back silently — which is
both convenient and dangerous. Put your fastest provider first and make sure it
*actually* loads (a silent fallback to CPU masks your "GPU" deployment).
```python
import onnxruntime as ort
ses = ort.InferenceSession("model.onnx",
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
sess_options=so)
print(ses.get_providers()) # confirm the first one loaded
```
Options include `CPUExecutionProvider`, `CUDAExecutionProvider`,
`TensorRTExecutionProvider` (NVIDIA, via the TensorRT package; needs graph opt
disabled and INT8 declared), and mobile/ARM providers. Match providers to where
you deploy.
## Graph optimizations
ORT applies graph-level transforms: node fusions, layout optimizations,
constant folding, kernel specialization. These are **leveled** (basic/extended/l
l/...):
- Run **online** (at session init) — simple, but repeats work on every init, which
hurts startup for complex models.
- Or **offline**: run the optimizer once and save the optimized graph to disk so
session startup is fast and consistent in production.
```python
so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
```
## Quantization (the big CPU win)
Dynamic or static integer quantization (FP32 → INT8) shaves size and speeds CPU
inference considerably, at some accuracy cost:
- **Dynamic quant**: easy, no calibration data, good speedup, small accuracy hit.
- **Static quant**: needs a calibration dataset and typically better accuracy at
the cost of more setup.
- Guardrail: quantize the biggest ops first, validate accuracy on a real slice of
traffic, and step back to per-channel or partial quantization if metrics dip.
## Stop paying the CPU↔GPU copy tax
ORT copies inputs from CPU even if the tensor is already on the device, and
copies outputs back — data movement that can make decoding *slower* than eager
PyTorch on GPUs. **IOBinding** pins inputs/outputs to device buffers to avoid
these copies. This is essential for low-latency GPU decoding workloads.
## Session laziness, warm-up, reuse
Inference sessions are expensive to create — **create once, reuse everywhere**.
Warm the session (a throwaway forward pass) before serving so the first real
request doesn't pay model-load + graph-opt latency. For concurrent requests,
micro-batching (batch 2–8) can pack GPU work better than pure singletons.
## Pitfalls
- Silent provider fallback (your "GPU" is actually CPU). Always print
`get_providers()`.
- Repeated session creation per request → massive startup overhead.
- CPU↔GPU copy overhead not addressed via IOBinding, especially in decoding.
- Blind full-model INT8 quantization that tanks accuracy — validate and
quantize selectively.
- Skipping warm-up, so the first request is slow.
- For TensorRT, forgetting to disable ORT graph optimizations (TensorRT does its
own).
## Verify
- Compare wall-clock latency (and p50/p95 under load) ORT vs the eager framework
on the same inputs.
- Confirm the intended execution provider is actually in `get_providers()`.
- Confirm INT8 accuracy is within tolerance on your validation set.
- Confirm session reuse + warm-up (first-request latency ≈ steady-state latency).