Skip to content

Performance results

The append-only record of comparative benchmark runs — what was measured, on which commit, in which environment. The numbers behind the scorecard in CANTALOUPE_PARITY.md live here; the harness contract is in specs/bench-harness.md. Per-run JSON/Markdown reports under bench/results/ are gitignored ephemera — this doc is the durable, curated record.

When you run the bench, add an entry (see Recording a run at the bottom). Don't rewrite past entries — they're a historical record of what that commit did on that hardware.

Environment (shared across the runs below unless noted)
Host AWS c7i.2xlarge — 8 vCPU (Intel Sapphire Rapids), 16 GiB, us-west-2
OS Ubuntu 24.04
Per-container limits --cpus 8 --memory 8g, identical for both servers
iiiris built from source per commit (distroless image; libvips 8.16 static, Go 1.25)
Cantaloupe 5.0.7 on Java 17 (Temurin); JP2 via Grok
Corpus master Van Gogh Irises, 9021×7122, Getty Open Content (sha256 e8db6a68…)
— large.tif vips tiffsave --tile --pyramid --compression jpeg --Q 90 → pyramidal tiled TIFF, 128 px internal tiles, ~104 MB
— master.jp2 OpenJPEG -n 6 -t 1024,1024 -p RPCL -r 20
Harness tools/bench-compare, --smoke --format jpeg,jp2,large
Metric caches-off saturation req/s (offered rate calibrated to ~100 % success); also cold (first-request) latency and peak RSS

Cantaloupe is a fixed reference (its numbers don't move with iiiris commits), so later runs are often iiiris-only (--servers iiiris) and compared against the Cantaloupe column recorded on 2026-06-07.

Results timeline — caches-off saturation req/s (iiiris)

Higher is better. Cantaloupe column is the 2026-06-07 measurement.

Cell Cantaloupe R1 baseline R2 mmap R3 +info-dims R4 +concurrency R5 +baseline-JPEG
info jpeg 82 71 512 512 512 512
info jp2 512 174 512 512 512 512
info large 512 18 512 512 512 512
full-scaled jpeg 2 9 10 10 10 15
full-scaled jp2 17 48 63 63 63 82
full-scaled large 23 17 238 238 238 176†
tiled jpeg 10 8 9 9 9 10
tiled jp2 41 35 40 39 53 85
tiled large 81 15 62 61 78 → 95 175

R4/R5 are max_concurrent: 16 (R1–R3 are 4). R5 also flips JPEG output from progressive to baseline, which lifts every JPEG-output render cell (the encode is ~3× cheaper). †full-scaled large dips because mc=16 over-subscribes that very fast op (cold 15 ms) — it's a tile-large tuning value; the cell still wins 176 vs 23. R3's win is in cost not throughput (info was already capped): see its note.

Runs

R1 — baseline (info-probe OOM fix)

  • Commit 7c663c6 · Date 2026-06-07 · Config max_concurrent: 4, both servers
  • What it isolates the OOM fix alone — the clean reference. The full caches-off matrix ran with 0 OOM / 0 restarts (prior runs crashed 3×).
  • Notable RSS info large 991 MiB, tiled large 924 MiB (whole-source buffering).
  • Finding confirms the real losses are genuine, not crash artifacts: info large 18, tiled large 15, info jp2 174.

R2 — mmap source I/O

  • Commit 0c62679 · Date 2026-06-07 · Config max_concurrent: 4, both servers
  • What it isolates mmap'ing filesystem sources (resident memory now independent of source size).
  • Finding three ❌ cells flip to the 512 cap (info large/jpeg/jp2); full-scaled large 17 → 238; tiled large 15 → 62. RSS 10–150 MiB vs Cantaloupe 0.3–2.2 GiB everywhere. tiled large 62 vs 81 — iiiris wins cold-latency (60 vs 65 ms), throughput still capped by max_concurrent: 4.

R3 — header-only info dimensions + heap O(1) LRU

  • Commit ace9886 · Date 2026-06-07 · Config max_concurrent: 4, iiiris-only
  • What it isolates reading info.json dimensions from the header (no libvips); heap cache O(1) LRU.
  • Finding info throughput was already capped, so the win is cost: info jp2 CPU 13 → 3 %, RSS 150 → 10 MiB, cold 2.7 → 1.3 ms (OpenJPEG header parse gone). info is now ~3 % CPU / 10 MiB / sub-ms for every format. Heap O(1) LRU not visible at the 512 cap / small smoke cache.

R4 — concurrency sized to the cores

  • Commit c248b34 (run) + a max_concurrent: 16 probe · Date 2026-06-07 · iiiris-only
  • What it isolates max_concurrent on the I/O-bound (EBS) tiled large decode. Product default is now NumCPU; bench config tested at 8 then 16.
  • Finding the EBS corpus makes tile reads I/O-bound, so cores idle during reads until concurrency exceeds the core count:
max_concurrent tiled large req/s CPU
4 61 ~2 cores
8 78 ~7 cores (≈ Cantaloupe 81 — tie)
16 95 ~7.5 cores (beats 81)

Bench configs set to 16. iiiris is less CPU-efficient per tile than Cantaloupe (~13 vs ~22 req/s/core) but out-throughputs it by using more of the same 8 cores, at ~12× less memory. Scorecard: 0 losing cells.

R5 — baseline JPEG (matched-output)

  • Commit 4ed49d5 · Date 2026-06-07 · Config max_concurrent: 16, iiiris-only
  • What it isolates JPEG output flipped progressive → baseline (the govips default was progressive — multiple entropy-coding passes, ~3× slower encode). Both servers now emit baseline q90, so this is a true matched-output comparison.
  • Finding the encoder was the per-tile CPU gap (parity item 2):
  • tiled large 95 → 175 req/s; per-core 13 → 25 (beats Cantaloupe's 22 — iiiris now more CPU-efficient per tile); cold 60 → 26 ms (vs 65); RSS 88–156 MiB vs 1717.
  • every JPEG-output render cell rose: full-scaled jpeg 10 → 15 (cold 553 → 363 ms), full-scaled jp2 63 → 82, tiled jp2 53 → 85.
  • full-scaled large 238 → 176 — not a regression in capability; mc=16 over-subscribes that very fast op (it's tuned for tiled large). Still 7.6× Cantaloupe's 23.
  • 0 OOM. iiiris vs Cantaloupe on tiled large: 2.2× throughput, more CPU-efficient per tile, 2.5× faster cold, ~15× less memory.

Recording a run

After a comparative run, append a new ### Rn — <label> section with:

  1. Commit (git rev-parse --short HEAD of the build under test) and date.
  2. Config / environment deltas from the table above (instance type, max_concurrent, caches, which servers, any corpus change).
  3. Findings — the cells that moved and why, in 2–4 lines.
  4. Add a column (or update the relevant cell) in the Results timeline if the run changes a headline number.

Keep the raw bench/results/*.json only if a number is surprising and worth re-deriving; otherwise the curated entry here is the record. Never edit a past run's numbers — add a new run instead.