Annotate with ChatGPT or Claude

AI Assistant can discuss the current image, generate editable annotations and manage project labels. Choose an assistant in Settings → AI Assistant, then open its annotation button in the annotator, describe what you want and review the result on the canvas. This is an optional online workflow, separate from the local Human and AI modes.

From a request to reviewed annotations

Open an image or pause a video on the frame you want to annotate.
Click Annotate with ChatGPT or Annotate with Claude on the right of the top annotation toolbar, above Next. The name follows your selected assistant. Its button becomes selected; Human and local AI become unselected, and local prediction runs are paused.
Choose OpenAI API, ChatGPT via Codex, Claude API or Claude Code from the available assistants. Web offers the two API connections; desktop also offers the installed CLI connections. Complete the connection steps below.
Use this image or frame in AI Assistant is enabled by default on entry and when you move to another image or frame. Attached identifies the current media. Uncheck it to send text without the image; that choice survives edits and closing/reopening the panel on the same frame.
Ask, for example: “Find every car and use the Car label.” For segmentation: “Outline each person with the Human label.” For classification: “Classify this image using the project labels.”
Review native AI suggestions on the canvas and in the annotation list. Refine pending boxes or polygons through follow-up messages, or correct them with manual tools.
Use the existing Accept / Reject controls above the canvas. Accept saves through the regular annotation workflow; Reject removes the suggestions. The chat has no separate annotation-approval controls and explicitly reports when nothing was annotated.

Annotate a batch of still images

Connect in AI Assistant first. The fourth Auto-annotate card follows your selected assistant and stays disabled without a configured API key or signed-in desktop connection. Try a small request before a large job; storing a key alone does not verify quota or model access.
Open Auto-annotate from Media and choose the dataset, selection, filter or Semantic search results. Only unannotated images are included by default. Videos and native video-frame items are skipped.
Select the chosen assistant after Visual prompt and enter Instructions for every image. Create or update project labels in AI Assistant beforehand; a batch uses the existing schema.
Start sends each eligible image and its project labels to OpenAI or Anthropic, according to your selection. API runs up to two images concurrently; desktop Codex and Claude Code run one at a time. Your selected service account, billing and usage limits apply.
Follow progress, then choose Open first image or Review in gallery. Use the existing Accept / Reject controls on Pending AI Review drafts. No confidence score is invented, so score-threshold acceptance does not apply to assistant results.

An explicit empty result counts as No predictions. Invalid image responses are reported individually. Authentication, quota and rate-limit errors pause the remaining scope. Cancellation keeps completed drafts; reopen Auto-annotate to resume. The prompt, model, connection and label schema are fixed for the job. Restore them or discard the job record and start again if they change. Requests are not automatically retried.

Supported tasks and project actions

Task	Output and review
Object detection	Axis-aligned bounding boxes with project labels; editable native AI suggestions.
Instance segmentation	Editable polygons. Check boundaries carefully: a vision-language model proposes contours; it does not run SAM to calculate a mask.
Classification	Single-label, multi-label and hierarchical image classes. Ancestors are added as required. Accept replaces the saved classification; Reject retains it.
Video	The currently paused frame only. No whole-video auto-labeling or tracking through this chat.
Keypoints	You can discuss the image, but generating skeletons or keypoints is not supported.
Projects	Read project summaries and create supported single-task projects with a confirmation card.
Project labels	Create, rename, recolor, or delete labels with a confirmation card. Review deletion effects before confirming.

You can combine actions: “Create a Dinosaur label, then outline all dinosaurs in this image.” Confirm the label operation in the chat; the assistant can continue to annotation afterward. Renaming or recoloring keeps the label identity. Label deletion may affect existing annotations, so its consequences are shown for review.

Chat can refine its own pending geometry suggestions. It does not freely rewrite manual or previously accepted objects, locked objects, tracks, or suggestions from another source. Confirmed label changes are separate project operations and can affect annotations using those labels.

Choose your AI connection

Platform	OpenAI API / Claude API	ChatGPT via Codex / Claude Code
Web browser	Your API key; stored in this browser. Separate API billing.	Not available in the web build.
Windows desktop	API key stored in the OS credential store; native requests.	Connect the corresponding installed Codex or Claude Code CLI and its account.
macOS desktop	API key stored in the OS credential store; native requests.	Desktop integration implemented; verify CLI availability and your installed build.
Native iOS / Android	Unavailable.	Unavailable.

OpenAI API

Select the OpenAI API card in AI Assistant, save your own API key and use Test API connection to check model access. You can also manage the key under Settings → AI Assistant.
Use an image-capable model with structured tool support. Model access and billing belong to your OpenAI account; ChatGPT subscription usage does not cover API requests.
The connection test makes a short billable API request without project data, images or chat history. Saving a key alone does not prove that the model or quota is available.

Claude API

Choose Claude API under Anthropic in Settings → AI Assistant or in the drawer. Save your Anthropic API key in its connection panel and choose an offered model.
The OpenAI billing guide and Test API connection button do not apply to Claude. Try a small message to check access before attaching sensitive content or starting a batch; saving a key alone does not validate it.
Web saves this key in browser storage; desktop uses the OS credential store. The OpenAI and Anthropic keys are stored separately. Requests go to Anthropic and use the access and billing of that API account.

Claude Code on desktop

Install the Claude Code CLI, then choose Claude Code under Anthropic. This connection requires the desktop app; installing the Claude chat application alone does not provide the CLI.
Connect your CLI account. AnnotateIt checks the existing Claude Code sign-in and can start the CLI login flow. If needed, complete claude auth login in a terminal, then reconnect.
If the executable is not found, set its path in the connection settings. Use a model alias or model ID supported by that account. This route uses the CLI account and its limits, independently of the saved Anthropic API key.

Claude Code uses the existing CLI account on this machine. Signing out through this connection also signs that CLI out. It does not create an isolated AnnotateIt-only Claude login.

ChatGPT on desktop

Install Codex CLI or a Codex desktop installation containing the executable. AnnotateIt tries standard locations and PATH.
Choose ChatGPT, click Sign in with ChatGPT, complete the browser sign-in, and return to AnnotateIt. ChatGPT connected confirms the connection; account details appear when available.
If Codex is not found, open Connection settings, set Codex executable path and apply it. Model ID must refer to an image-capable model available to your account.

This route uses your eligible ChatGPT account’s Codex access and limits, not the OpenAI API key. AnnotateIt starts an isolated Codex session, stores authentication separately in the OS credential store and disables shell access. It is not a link to your ChatGPT conversation history.

Sources: OpenAI: Codex authentication OpenAI: usage and pricing

Store builds may lag behind the current source. Windows protocol checks and browser UI tests do not establish compatibility with every macOS or Mac App Store sandbox configuration; macOS needs device-specific validation.

What leaves your device

Data or action	Behavior
Chat messages and requested project context	Sent to OpenAI or Anthropic through your selected connection. Project tools can include names, labels, task types and dataset summaries.
Current image or paused video frame	Sent only with a message while the attachment checkbox is enabled. JPEG preview, at most 1600 px on its longest side, plus original dimensions.
Project labels and annotation context	Included with the attached-image request so proposals use the right schema and current scene.
Entire video or dataset	Not sent by the annotation chat. Only one image or frame is included; prior image attachments are removed from outgoing history.
AI Assistant batch scope	Starting Auto-annotate sends eligible still images in separate requests, with the common instructions, project labels and original dimensions. Each preview is at most 1600 px on its longest side.
Opening the panel or enabling attachment	Does not by itself send image pixels.
Local annotation and local AI tools	Run on your device independently of this optional online connection.

Navigation, errors and limits

Moving to another frame discards unaccepted suggestions and attaches the new frame in external-assistant mode. Playback clears the image context until a frame is available again.
Closing the drawer keeps the selected assistant active. Selecting Human or local AI leaves external-assistant mode, closes the drawer and clears image sharing.
If the image, labels or annotations change while a result is pending, an outdated proposal cannot be applied to the new scene. Send a new request.
No matches and failed annotation generation are reported in the chat; a conversational reply alone does not mean annotations were created.
Check polygons and class choices before Accept. This feature does not promise exact contours, bulk annotation, automatic video tracking or keypoint generation.