MobileSAM vs SAM 2.1 vs SAM 3: Which One Is Actually Faster for Annotation?

The Models page in AnnotateIt currently gives me five ways to answer one apparently simple request: I click an object, the app gives me its outline. MobileSAM is already there. SAM 2.1 arrives in Tiny, Small and Large. Then SAM 3 Tracker sits at the heavy end of the row looking like the obvious final boss. Naturally, I wanted a table that said which one was fastest and which one made the best mask.

That table should have taken an afternoon to find. It did not. I found official speed claims, model papers, runtime ports, medical comparisons, a fire-segmentation study and enough tables to make a spreadsheet feel self-conscious. None put the five engines AnnotateIt actually runs through the same browser, on the same device, with the same input and stopwatch. So we ran that part ourselves.

The result settles the local runtime question on the machines we tested. It does not settle which model saves the most annotation time, because our harness measures encoders, decoders and memory — not how many corrections a mask needs before a person accepts it. This article keeps those two finish lines separate: first the race we completed, then the quality study that still needs doing.

The short answer, if you need to pick one now

You want zero setup, use a phone or tablet, or work in the desktop app

Use MobileSAM. It is built in, works on every platform and was the fastest engine we measured: 72 ms on the M4 Max GPU.

You have WebGPU and want a downloadable step above the built-in model

Try SAM 2.1 Tiny. Its 150 ms encoder was about twice MobileSAM’s time, but still comfortably interactive.

You are running without WebGPU

Stay with MobileSAM unless a real mask gives you a reason to move. It led the CPU/WASM rows too: 527 ms multi-threaded and 1.93 s single-threaded on the Mac.

Tiny repeatedly misses a boundary you care about

Test Small, SAM 3 Tracker or Large on that exact object. The benchmark tells you the extra wait, not whether the new mask earns it back.

You want full SAM 3 concept prompting — text or an example finding every matching object

That is a different capability. AnnotateIt currently exposes a community ONNX build of the SAM 3 image tracker as an interactive click engine, not the full upstream concept detector.

That recommendation is intentionally less exciting than a podium. It is also what I would do with my own dataset: start small, move only when an actual mask gives you a reason, and judge the cost in corrections rather than model prestige.

The first trap: everybody brought a different stopwatch

MobileSAM’s official repository says the whole pipeline takes about 12 ms on a single GPU: 8 ms for its TinyViT encoder and 4 ms for the shared SAM decoder. The repository does not name that GPU in the table. It also mentions roughly three seconds for a CPU demo on the authors’ Intel Mac. Both numbers are useful descriptions of their implementation. Neither predicts what the ONNX model will do inside your browser.

Meta reports 91.2 FPS for SAM 2.1 Tiny, 84.8 for Small and 39.5 for Large on an A100 with compiled PyTorch. Those are impressive video-throughput figures on a data-centre GPU. They are not image-encoder latency in onnxruntime-web, and they are definitely not the pause you see after switching photos in a local annotation tool.

SAM 3 makes the comparison stranger again. Its headline result is a two-times gain on promptable concept segmentation, a task where text or an image example asks the model to find every matching instance. SAM 3.1 then reports about a seven-times speed-up at 128 tracked objects on one H100. Neither claim is “one positive click on one still image in the browser.” They answer different questions.

Sources: MobileSAM — official repository and timings · Meta — SAM 2.1 checkpoints and A100 benchmark · Meta — SAM 3 paper · Meta — SAM 3.1 release notes

“Faster” without the hardware, runtime, input, warm-up rule and timed boundary is not a result. It is a loose adjective wearing running shoes.

What we actually measured inside AnnotateIt

The completed benchmark runs every interactive engine AnnotateIt offers: the built-in MobileSAM, SAM 2.1 Tiny, Small and Large, and the exact SAM 3 Tracker build shipped by the product. Same 16-inch MacBook Pro (2024, Mac16,5) with M4 Max, same Chrome and onnxruntime-web versions, same real photograph, same product weights and session settings, and a fresh browser profile for every cell. Each encoder median comes from eight timed runs after a discarded warm-up; each decoder median comes from sixteen clicks.

Browser RAM delta is the growth in Chrome’s complete process-tree resident set from the loaded benchmark page to the end of the run. On macOS that figure includes shared pages. It is not peak memory, total system memory or separate GPU memory.

Those are the WebGPU rows, ordered by encoder time. The machine was a 16-inch MacBook Pro (2024, Mac16,5) with an Apple M4 Max (16-core CPU, 40-core integrated GPU) and 48 GB of unified memory, running Chrome 151 and onnxruntime-web 1.24.3 on macOS 26.5.2. Measured on 14 August 2026. This is a fast laptop, not a promise about yours.

The clean result is MobileSAM. At 72 ms it was the fastest WebGPU encoder, the fastest multi-threaded WASM encoder at 527 ms and the fastest single-threaded encoder at 1.93 seconds. Tiny is the only downloadable model in the same interactive neighbourhood: 150 ms on WebGPU, almost exactly matching the 154 ms we measured earlier on an RTX 3090.

Small and Large do not get the same WebGPU gift. Their quantised operators likely fall back heavily to WASM and pay transfer overhead, although proving that cause would require the per-operator profile this harness does not collect: Small takes 1.92 seconds, Large 10.26. SAM 3 Tracker lands between them at 3.29 seconds and has no CPU fallback. Once any image is encoded, though, the model choice barely changes the interaction — every WebGPU click in the table takes 12–42 ms. Large charges ten seconds at the door, not ten seconds for every correction.

Sources: AnnotateIt — complete five-engine benchmark, full backend matrix and method

What the five-engine run proves — and what it cannot

We can now compare local runtime honestly. On this M4 Max, in the same browser harness, MobileSAM opens the image first, Tiny is close enough to feel instant, Small crosses into a noticeable pause, SAM 3 Tracker takes about three seconds and Large takes about ten. We can also say that every decoder remains responsive after the image is ready, and we know the memory cost each engine asks the browser to carry.

What we cannot say is that the fastest encoder makes the fastest finished annotation. The harness sends a point and times the returned mask; it does not own a reviewed ground-truth mask, count the next correction or watch a person decide the boundary is good enough. It measures the machine’s part of the wait, not the human’s part of the job.

The SAM 3 name still needs care too. Upstream SAM 3 is a larger system with a detector, a tracker and concept prompts. The engine timed here is AnnotateIt’s SAM 3 Tracker: a pinned community ONNX conversion of the image tracker, wired to the same positive-and-negative-click workflow as the other interactive engines. It is not the full concept detector, and it is not SAM 3.1. The 3.29-second result belongs to that exact product engine — which is precisely why it is useful and precisely why it should not borrow another implementation’s quality claims.

The neighbouring research still matters. A broad SAM survey maps the model family; domain papers compare MobileSAM with SAM 2.1 variants on fire images and compare SAM 2 with SAM 3 in medical segmentation; robotics work tests lightweight segmenters under its own constraints. They tell us where quality can change by domain. Our runtime table tells us what those five buttons cost locally. Neither replaces the missing correction study.

Sources: Survey of Segment Anything models · MobileSAM and SAM 2.1 variants on promptable fire segmentation · SAM 2 versus SAM 3 in medical segmentation · Lightweight segmentation in robotics · sam3.cpp community runtime benchmarks

Raw latency is the wrong finish line anyway

MobileSAM opens an image in 72 ms and Large takes 10.26 seconds. Easy win for MobileSAM. Now suppose its first mask needs four corrections around bicycle spokes and Large needs one. The decoder time is almost irrelevant; the human hand and eyes are not. Which model was faster?

The only answer worth publishing is time to accepted annotation: the clock starts when the next image is requested and stops when a person says the mask is good enough for the dataset. That total includes session start, image encoding, the first click, every positive or negative correction, manual brush work and the tiny pause where you inspect the boundary and decide whether you trust it.

Interactive-segmentation research already has better ideas than “one IoU after one lucky click.” Number of Clicks counts how many prompts are needed to cross a target overlap such as 85, 90 or 95 percent. Boundary scores catch the thin edges that a broad overlap number can hide. RClicks shows why click simulation has to resemble the corrections people actually make. Those are the bones of a serious model-quality test. I would add the wall clock and the acceptance decision, because that is where an annotation tool either saves your afternoon or merely wins a paper table.

Sources: Rethinking Interactive Image Segmentation — latency, quality and NoC · RClicks — realistic click simulation

The next experiment: which engine saves human time?

The runtime phase is done. The useful second phase is a quality-and-correction study, and AnnotateIt is unusually well placed to run it because all five engines already sit behind the same tool. No custom demo for one model, no separate post-processing script for another, no server latency wandering into the result.

1. Choose 200 objects: 50 ordinary photographs, 50 industrial objects, 50 medical or scientific shapes that can be shared legally, and 50 thin, occluded or edge-heavy objects picked specifically to be annoying.
2. Create a reviewed ground-truth mask for every object and publish the inclusion rules. “Good enough” has to mean something before anybody sees which engine produced the mask.
3. Reuse the exact MobileSAM, SAM 2.1 Tiny, Small, Large and SAM 3 Tracker builds from the completed runtime benchmark. Record model hashes with every mask, not just names.
4. Give every engine the same first positive point. Then use a deterministic correction policy for the full set, with a smaller blinded human pass to check that the simulated clicks still behave like a person’s corrections.
5. Join the measured session, encoder and decoder timings to each quality run, keeping first-image cost separate from steady-state cost.
6. Report first-click IoU, boundary F-score, NoC@85, NoC@90, NoC@95, failure rate and total time to accepted annotation. Break every result down by domain rather than hiding a disaster in one category inside a cheerful average.
7. Publish all individual samples and failed cases. A median without its ugly masks is how a benchmark becomes a brochure.

The five-engine runtime study is complete. The five-engine quality and human-time study is not. Keeping those two sentences separate is the difference between a benchmark and a brochure.

My hypothesis, clearly labelled as a hypothesis

I expect MobileSAM or SAM 2.1 Tiny to win ordinary objects on total time, because there is not much quality left to buy after the first clean mask and their opening cost is low. I expect Small or Large to earn back some of their wait on hair, spokes, wires, holes and cluttered boundaries. I do not know whether the SAM 3 Tracker conversion will win enough corrections to pay for its weight. That last sentence is the reason to run the test.

I also expect the best downloadable choice to change by platform. Tiny beats Small on multi-threaded WASM on the M4 Max (1.60 versus 1.98 seconds), while Small edged Tiny on the Intel desktop (1.68 versus 1.88). MobileSAM wins the raw latency race on every Mac backend we ran; the interesting question is where another model earns its wait back in fewer corrections.

The bottom line

On the M4 Max, in the actual local browser runtime, MobileSAM is the clear speed winner at 72 ms per image. Tiny is next at 150 ms, then Small at 1.92 seconds, SAM 3 Tracker at 3.29 and Large at 10.26. Every one responds to correction clicks in 12–42 ms once the image is ready. Those are no longer estimates or numbers borrowed from somebody else’s GPU; they are the five product engines in one harness.

MobileSAM is therefore the sensible default because it is there, light and fast everywhere we measured. Tiny is the inexpensive experiment when a mask needs more help. Small, SAM 3 Tracker and Large need to win back visible opening waits with better boundaries — something latency alone cannot prove. And none of us should call a model the annotation winner until it has raced to an accepted mask, not merely to the end of an encoder call.

So my answer is: start with the smallest engine that handles your real objects, keep the one that needs the fewest seconds of your attention, and be suspicious of any benchmark that cannot tell you exactly what its stopwatch included. I went looking for a winner. I came back wanting a better race.