Text-prompt detection with Grounding DINO
Type a description — "forklift", "helmet", "red car" — and an open-vocabulary detector proposes boxes for review. Grounding DINO Tiny is a local WebGPU-only download offered in Windows, macOS and compatible desktop browsers with a working adapter; it is absent on mobile.
What open-vocabulary detection buys you
A normal detector only knows the classes it was trained on. An open-vocabulary detector takes your words as the query, so you can ask for an object class that was never in any training label set of yours. In practice that removes the worst part of starting a new detection project: the first hundred boxes, drawn by hand, before any model of yours exists to help.
It is an assistant, not an oracle. Proposals are reviewed and accepted or rejected — they are not silently written into the dataset.
What it needs
| Requirement | Detail |
|---|---|
| Engine | Grounding DINO Tiny, downloaded once from the Models page and then cached locally |
| Runtime | WebGPU — the engine is not offered where WebGPU is unavailable |
| Platforms | Windows, macOS and the web app in a compatible desktop browser with a working WebGPU adapter. Not offered on iPhone and iPad |
| Network | Only for the one-time download; inference is local |
Grounding DINO Tiny is the only engine under the Text prompt capability on the Models page, and it is a download: an open-vocabulary detector that predicts boxes directly from your words. It is not the only way to turn a phrase into shapes, though. The default CLIP engine (or SigLIP 2) sits under Zero-shot and powers the annotator’s own Text prompt tool. Whichever interactive-segmentation engine is active — MobileSAM, a SAM 2.1 variant or SAM 3 Tracker — proposes candidate regions, and CLIP or SigLIP 2 ranks each against the prompt. The toolbar names the exact pair it will run. Its displayed Match value is a similarity score on that method’s scale, not a probability that the annotation is correct.
Using it
- Open an image in a detection project and select the text-prompt tool.
- Type what you are looking for. Short, concrete noun phrases work best — "forklift" beats "vehicle used for lifting".
- Review the proposed boxes. Accept the good ones, reject the rest.
- Adjust anything that is close but not right — accepted proposals are ordinary annotations you can drag and resize.
The other ways to get boxes
- By hand — still the right answer for the images where precision matters most.
- The detection assistant — box one example and every region of the same image that resembles it comes back as a proposal. A similarity routine rather than a model, so it needs no download and is at its best on a picture full of repeats.
- Find by example — a prediction source rather than a toolbar tool. It uses the objects you have already annotated as references and looks for similar ones on the image you are on, running on the active segmentation engine.
- A curated Auto-annotation detector — download a ready-to-use COCO-80 model (D-FINE, RT-DETR, RT-DETRv2, DEIM, RF-DETR or EdgeCrafter ECDet) and set it up in the project. No training or ONNX file of your own is needed, as long as your objects are among the 80 COCO classes.
- Your own model — import a trained YOLOv8, RT-DETR or DETR as ONNX and pre-label with it. Experimental.