MobileSAM vs SAM 2.1 vs SAM 3: a local benchmark

Two machines, five engines. These figures describe the two devices in the tables below and nothing else — a different laptop, an older phone or another browser will differ, sometimes by a lot. That is the reason the harness ships with the numbers: node scripts/benchmark-sam2.mjs downloads (or, for the bundled MobileSAM, reads) the same weights, drives the same page and prints the same tables for whatever you run it on.

First: “CPU”, “WASM” and WebGPU are not three things

A benchmark that lists CPU and WASM as separate rows is measuring something that does not exist here. AnnotateIt runs models through onnxruntime-web, where the execution provider named cpu IS the WebAssembly backend — there is no native CPU path in a browser. So there are two backends, not three, and the meaningful third row is a difference in threading:

That last row is not a hypothetical. Cross-origin isolation is what makes SharedArrayBuffer available, SharedArrayBuffer is what lets ORT spawn threads, and a WebView that does not provide it drops straight to one thread. It is the single largest performance factor in the whole table — larger, for two of the three models, than whether there is a GPU involved.

Five engines, three families

AnnotateIt runs five interactive Segment Anything engines on your own hardware, and they do not behave alike — they come from three different model families with three different runtime contracts. That is why one benchmark table cannot be read as a single ranking:

Three differences in that table do most of the explaining. MobileSAM is a tiny distilled encoder, so it is fastest everywhere and it is the only engine whose decoder also runs on the GPU. The SAM 2.1 and SAM 3 decoders are pinned to the CPU provider because ORT-web’s WebGPU backend executes those particular graphs incorrectly — which, as the tables show, costs nothing, because the decoder is cheap. SAM 2.1 Large is WebGPU-gated for availability but retains the same CPU session-creation fallback as the other SAM 2.1 encoders. SAM 3’s full-size encoder graph could technically execute on the WASM CPU provider, but the product deliberately provides no CPU fallback because that path can take minutes and exhaust memory.

Results — Apple M4 Max

Encoder and decoder are timed separately because they happen at different moments. The encoder runs once when you open an image; the decoder runs on every click. The encoder figure is the median of 8 timed runs and the decoder figure the median of 16, both after a discarded warm-up, and both include reading the result back out of the runtime — on WebGPU a run that has not been read back has not necessarily finished. This first table is a 16-inch MacBook Pro (2024, Mac16,5) with M4 Max; the Intel/RTX 3090 desktop follows below.

SAM 2.1 Large’s WASM rows are reference measurements from the benchmark harness. The product offers Large only behind its WebGPU capability gate, but its session can fall back to the CPU provider if WebGPU/JSEP initialisation throws. SAM 3 has only a WebGPU row because the product hard-pins its encoder to WebGPU and does not expose the technically possible but impractically slow, memory-heavy WASM path. The macOS Browser RAM delta is the growth in the resident set of Chrome’s process tree, which counts shared pages, so it reads higher than the Windows table’s private-working-set delta further down; treat the two machines’ RAM columns as internally comparable, not comparable to each other. Neither column is a peak-memory measurement.

Run-to-run spread was tight enough that these medians are meaningful rather than lucky: MobileSAM on WebGPU ranged 71–73 ms across its 8 runs, SAM 2.1 Tiny on WebGPU 148–150 ms, SAM 3 on WebGPU 3.27–3.46 s. The one exception is SAM 2.1 Large single-threaded, which hit the harness time budget after 3 runs rather than 8; its spread over those three was 37.24–37.27 s.

What the numbers actually say

MobileSAM is the fast default, and the numbers say why

The built-in engine encodes an image in 72 ms on WebGPU, 527 ms on multi-threaded WASM and under two seconds even on a single thread — the fastest engine within each measured backend. It is a small distilled encoder, so it is also the only engine whose decoder runs on the GPU (12 ms per click) rather than being pinned to the CPU. This is exactly why AnnotateIt ships it as the always-available default on phones, tablets and both packaged desktop apps; packaged Windows can also use its multi-threaded WASM worker pool.

WebGPU transforms SAM 2.1 Tiny and does nothing for the quantized sizes

Tiny goes from 1.60 s on multi-threaded WASM to 150 ms on WebGPU — an 11× speed-up, and 40× against the single-threaded case. Small and Large get no such gift: Small is fractionally faster on WebGPU than on multi-threaded WASM (1.92 s against 1.98 s is effectively a wash), and Large is actually slower on the GPU here (10.26 s against 9.60 s).

The likely reason is weight format rather than size — with the caveat that this table cannot fully separate the two, because Tiny is both the only fp32 model and the smallest one. Tiny ships as fp32; Small and Large are quantized to 8-bit integers to keep them downloadable, and integer-quantized operators are poorly served by the WebGPU backend, so much of those graphs falls back to WASM anyway and what remains pays to move data between the two. Settling it would take a per-operator breakdown of what the backend actually accepted, which this harness does not collect. The lesson survives either way: “has a GPU” is not the question, “can this backend execute this graph” is.

SAM 3 is the heavy, WebGPU-only option

The SAM 3 Tracker encodes in 3.3 s and runs only on WebGPU — there is no WASM fallback, so on a machine or a WebView without it the engine is simply unavailable. That is the runtime cost of this heavier tracker encoder: it takes more than twice Tiny’s WebGPU encode and over forty times MobileSAM’s. Its per-click decoder, pinned to the CPU like SAM 2.1’s, is still a comfortable 42 ms. This benchmark does not measure whether its masks need fewer corrections.

The click is fast in every configuration

Wherever a GPU or threads are available the decoder sits between 12 and 46 ms — MobileSAM lowest because it runs on the GPU, SAM 2.1 and SAM 3 in the high twenties to mid forties on their pinned CPU provider — and 86–104 ms single-threaded. SAM 2.1 and SAM 3 pin the decoder to the CPU on purpose: ORT-web’s WebGPU backend executes those graphs incorrectly (it returns a fixed speckle mask with near-zero confidence wherever you click), and pinning it costs nothing measurable. So the interactive experience is the same shape for every engine: a wait when the image opens, then clicks that feel instant. Only that first number moves by two orders of magnitude across engines and backends.

Results — Intel NUC13 / RTX 3090

The same harness on a fast Windows desktop with a discrete NVIDIA GPU. Only SAM 2.1 was measured on this machine; MobileSAM and SAM 3 rows exist for the Mac above. The Browser RAM delta here is the growth in the Windows private working set, which is why its figures read lower than the macOS resident-set deltas above.

An integrated Mac GPU matches a discrete RTX 3090 — until the graph is quantized

For the fp32 Tiny encoder the two machines are a dead heat on WebGPU: 150 ms on the M4 Max’s integrated GPU against 154 ms on the RTX 3090. Apple’s unified-memory GPU is doing, for this workload, what a 350-watt discrete card does — and MobileSAM on the same Mac GPU is faster still at 72 ms. Where the Mac falls behind is the single-threaded WASM column on the quantized graphs: SAM 2.1 Large takes 37 s on one Apple thread against 25 s on one Intel thread, because ARM WASM SIMD serves those 8-bit-integer operators less well than the Intel path does. The GPU story favours parity; the pure-CPU story still favours the big x86 desktop.

Memory

Two different measurements, because they answer two different questions. The figures below are from the M4 Max.

• Page memory is what the browser attributes to the page itself. On WebGPU it is small because the weights live in GPU memory rather than in the WASM heap — 144 MiB against 820 MiB for the same SAM 2.1 Tiny model.
• Browser RAM delta is the growth in the whole browser process tree from the moment the page is loaded to the end of the run. On macOS this is resident set (which counts shared pages); on Windows it is the private working set. It includes memory staged on the host for GPU execution, which is why the WebGPU rows can be larger even when their page memory is smaller. It is not peak memory, total system memory or separate GPU memory.

Video memory is missing from these tables on purpose. On the Windows/NVIDIA box, nvidia-smi reports per-process usage only for compute contexts, and a browser’s WebGPU work is a graphics context, so the figure came back as zero — wrong rather than merely absent. On the M4 Max there is no separate video memory to report at all: the GPU shares the machine’s unified memory, whose host allocations contribute to the Browser RAM delta. Rather than publish a system-wide delta that also moves when anything else touches the GPU, the column is left out.

The single-threaded rows have no page-memory figure at all: the browser API that reports it requires cross-origin isolation, which is precisely what that configuration lacks. Their host-RAM figures are measured normally.

The devices

Both are fast machines with capable GPUs, which makes them the optimistic end of the range rather than typical ones. Read the single-threaded column as the pessimistic end. A slower laptop, an older phone or a machine without WebGPU moves every row — usually all in the same direction, but as the quantized single-threaded numbers show, not always by the same amount.

Method

• The weights are the ones the product uses: SAM 2.1 fetched from models.annotateit.ai and SAM 3 from the pinned Hugging Face revision the app downloads it from, both verified against the SHA-256 checksums the app ships; MobileSAM is the copy bundled inside the app. No local conversion, no re-export.
• The harness uses the product’s graphs, graph optimisation, SIMD and execution-provider wiring: SAM 2.1 and SAM 3 decoders are pinned to CPU, MobileSAM’s decoder stays on the encoder provider, and SAM 3’s encoder is WebGPU-only in the product. The single-thread reference cells retain the published browser harness’s parallel graph-execution mode; packaged macOS selects sequential execution when its WASM runtime has one thread.
• Pre-processing mirrors each engine’s own configuration — 1024×1024 for MobileSAM and SAM 2.1, 1008×1008 for SAM 3, with each model’s own normalisation — and is timed separately; it is not part of the encoder figures. Because only tensor shape and dtype affect latency, the canvas resample standing in for the app’s OpenCV filter does not move the numbers.
• The image is a real photograph from the sample datasets, not noise.
• The decoder is timed with a single positive click in the centre of the image, 16 runs per cell, in each engine’s own decoder contract.
• Each cell runs in a freshly launched browser with its own profile, so nothing carries over between them.

node scripts/benchmark-sam2.mjs                            # SAM 2.1 Tiny / Small / Large
node scripts/benchmark-sam2.mjs --variants mobilesam,sam3  # the built-in engine and SAM 3
node scripts/benchmark-sam2.mjs --variants tiny            # a single engine

The harness runs on macOS, Windows and Linux; host details and per-process memory are read the native way on each. The raw output, including every individual sample rather than just the medians, is written to scripts/sam2-benchmark/results.json (or the --out path you pass).

Which engine to actually use

Whichever you pick, the image never leaves the device. Everything in this table is the cost of doing the work locally instead of sending your pictures to somebody’s server — which is the trade the product exists to make.