Connect a runner for fine-tuning and inference
Pipelines is an experimental integration with a separately operated ML runner. The current recipe targets NVIDIA TAO RT-DETR for axis-aligned image object detection. AnnotateIt prepares a frozen dataset and manages job requests; Docker and GPU execution happen in the runner environment.
Where it works
| Environment | Current support |
|---|---|
| Current native desktop build | Native HTTP transport to a compatible HTTPS runner, or the local pilot runner at http://127.0.0.1:8317. |
| Hosted web app | Runner connections are blocked by the current transport policy. |
| Local browser development build | Only http://localhost:8080 or http://127.0.0.1:8080 to http://127.0.0.1:8317, with matching runner CORS configuration. |
| Mobile | No ML runner transport. |
Prepare the runner first
Operate a compatible ML Backend API v1 runner with the versioned tao-rtdetr recipe, a prepared TAO container and checkpoint, Docker with a Linux engine, and suitable NVIDIA GPU access. The app does not install these prerequisites. Check connection and readiness before creating a training job. A reachable HTTP service alone does not prove Docker or GPU readiness.
This integration requires a separately verified runner environment. Successful connection checks do not establish that a GPU training job can complete; validate an end-to-end run in your environment before relying on it.
From a dataset version to a prediction preview
- In Settings → ML connections, enter the runner address and its session token, then check the connection. The token is kept in memory and must be supplied again after restart.
- Use an ordinary Object Detection project with reviewed PNG/JPEG images, explicit empty labels for verified negatives, and saved training and validation splits.
- In Pipelines, select the connection, exact recipe version and a saved dataset version, or explicitly create a version. Start fine-tuning sends its COCO bundle to the runner.
- Inspect saved runs, refresh status and logs, retrieve artifacts or request cancellation. Closing the page does not stop a backend job. Retry a saved run after an uncertain network outcome rather than creating a duplicate.
- After training completes, select its checkpoint and an image dataset version for inference. View predictions displays validated boxes, labels and confidence alongside the source image.
Your runner receives the selected data
Starting a job transfers the selected dataset bundle, including media and annotations required for training, to the configured runner. A runner on your own machine keeps execution there; a remote runner receives those bytes. This is separate from the on-device annotation models and is an explicit exception to a local-only workflow.
What this integration does not do yet
- The current training recipe does not convert rotated boxes, segmentation, keypoints or videos into ordinary detection data.
- The trained model stays inside Pipelines; registration in the general Models tab is not wired up.
- Runner prediction previews do not modify manual annotations and cannot yet be accepted into Pending AI Review.
- There is no hosted deployment endpoint, automatic retraining or universal recipe support. Native store releases must include this integration before their installed build can use it.