Skip to content

Filesystem cache backend

The contract iiiris maintains for the filesystem cache backend across both its modes (single-host local + multi-replica distributed via a Redis coordinator). This spec replaces docs/briefs/FS_CACHE2.md now that the work has shipped end-to-end.

The filesystem backend is one of four storage cache backends iiiris ships (heap, filesystem, s3, redis — plus none, which disables a slot rather than storing anything) and is the default choice for single-binary deployments that want persistent caching without an external object store. It is safe to share between concurrent processes on one host (local mode), and safe to share between multiple replicas on one shared NFS/EFS mount when a Redis coordinator is configured (distributed mode).

Modes

Mode When Index lives in Use cases
Local coordinator: block absent (default) The filesystem itself (stat for size, mtime for LRU), serialized by flock on <root>/.lock Single-process; concurrent processes on one host (blue/green restarts, sidecar tools).
Distributed coordinator.redis.addr set Redis (HASH for sizes, ZSET for LRU, counter for total bytes, TTL'd lease for eviction) Multi-replica iiirisd on a shared NFS/EFS mount.

Mode is chosen at boot from config.CacheConfig. There is one Go backend type per mode (*Filesystem and *FilesystemRedis); both satisfy RenderCache, InfoCache (via the small FilesystemInfo / FilesystemRedisInfo wrappers), OriginCache, Stater, and Purger.

Surface

Public Go API

Constructors (used internally by cache.BuildRender/BuildInfo/BuildOrigin):

  • cache.NewFilesystem(root string, maxBytes int64) (*Filesystem, error)
  • cache.NewFilesystemRedis(ctx context.Context, cfg config.CacheConfig) (*FilesystemRedis, error)

Methods (both modes — same signatures):

Get(ctx, key) (Entry, error)            // ErrMiss on absent
Put(ctx, key, contentType, body) error  // tmp+rename
Delete(ctx, key) error
Name() string
Stats() Stats
PurgeAll(ctx) error
EvictNow(ctx) error                     // explicit synchronous evict pass

Wrappers (one per role × mode, four total):

FilesystemInfo / FilesystemRedisInfo    — InfoCache.Put ([]byte)
FilesystemOrigin / FilesystemRedisOrigin — OriginCache.Put (no contentType)

Config

YAML under caches.<role> (where <role> is one of render, info, origin):

backend: filesystem
path: /var/cache/iiiris/render   # required
max_bytes: 5368709120            # 0 = unbounded
coordinator:                     # absent → local mode
  redis:
    addr: redis:6379
    password_env: REDIS_PASSWORD
    db: 0
    tls: false
    prefix: iiiris               # optional Redis-key namespace prefix

coordinator.redis.addr set with any non-filesystem backend is rejected at boot. backend: redis set together with coordinator.redis.addr is rejected at boot (the coordinator promotes the filesystem backend; it doesn't combine with redis-as-backend).

On-disk format

Sharded by SHA-256 prefix:

<root>/<aa>/<bb>/<full-64-char-sha256-hex>

Plus two control files at the cache root:

<root>/.iiiris-cache-version     // migration sentinel; contents: "v2\n"
<root>/.lock                     // flock target (local mode only)

In-flight writes appear as <root>/<aa>/<bb>/tmp-* until renamed.

Redis keyspace (distributed mode)

All under <prefix>fscache:<8-hex-of-sha256-of-absolute-cleaned-root>: where <prefix> defaults to iiiris: (from coordinator.redis.prefix, with a : appended if non-empty).

entries  HASH    hash → size_bytes (decimal string)
lru      ZSET    hash → mtime_unix_nano (score)
total    STRING  total bytes counter (decimal string)
lease    STRING  eviction lease holder id (with TTL = 30s)

The 8-hex namespace is derived from the absolute cleaned cache path, so multiple cache instances against the same Redis (one per role, or even across deployments) use distinct keyspaces without operator-side namespace coordination.

Contracts

  1. Get and Put are lock-free in both modes. Only the eviction critical section acquires the coordination primitive (flock in local mode, SET NX EX Redis lease in distributed mode). Put is os.CreateTemp + io.Copy + os.Rename; Get is os.Open plus a best-effort LRU touch (os.Chtimes locally, ZADD in Redis).

  2. Eviction is serialized. At most one process per cache root runs an eviction pass at a time. The opportunistic Put-side trigger (tryEvict, fired when a per-process bytes-written counter crosses MaxBytes/100) uses a non-blocking TryLock / SET NX — if the coordination primitive is held, the trigger silently skips and the holder's pass catches up.

  3. Mtime is the LRU clock. Local mode reads it via stat; distributed mode reads the ZSET score. Get refreshes the clock on hit; Put sets it on write. Filesystems mounted noatime are unaffected — mtime, not atime, is the source of truth. When two entries share an mtime (sub-resolution writes on a coarse-mtime filesystem, or simultaneous writes from different replicas), eviction order between them is unspecified.

  4. Cap counts only hash-named files. MaxBytes is enforced against files matching ^[0-9a-f]{64}$. Operator-dropped files (READMEs, .DS_Store, the control sentinels above, in-flight tmp-* files) do not count toward the cap and are never selected as eviction victims.

  5. Operator files are preserved across every lifecycle event. PurgeAll, eviction, migration, and reconciliation only touch files that match the hash-name shape or the tmp-* orphan sweep pattern (older than 1 hour). An operator who drops a README or a 5 GB tarball into the cache root will find it untouched after any iiiris operation.

  6. Cap enforcement is best-effort across processes/replicas. In local mode the on-disk total may briefly exceed MaxBytes by up to write_rate × walk_interval between eviction passes (typical overshoot <1%). In distributed mode the bound is tighter because Redis maintains a strongly consistent counter; overshoot is bounded by writes-in-flight when eviction starts.

  7. The eviction pass survives the lock-free-Put race. Each victim's mtime is re-checked immediately before os.Remove; if mtime moved since the walk (a concurrent Put or Get-touch refreshed it), the victim is skipped. Preserves the lock-free Put / Get-touch property without corrupting freshly-written entries.

  8. Orphan tmp-* files are swept after 1 hour. The eviction walk removes any tmp-* file with mtime < now-1h. The grace window is comfortably past any realistic libvips render time (seconds at worst) so live in-flight writes are never targeted.

  9. Local mode is single-host. The local-mode flock does not work reliably over NFS; multi-replica deployments on a shared mount MUST use distributed mode. iiiris does not refuse to start in this misconfiguration (the runtime sentinel was deferred — see Out of scope); operators are warned in the docs and via the mode=local (single-host only) startup log line.

  10. Distributed mode reconciles on startup. Each replica acquires the lease at construction time, walks the disk, and aligns the Redis index: orphan files (on disk, not in index) are added with their current mtime; ghost entries (in index, missing from disk) are dropped. Concurrent replica startups serialize through the lease; one reconciles and the others trust the result.

  11. Multi-key Redis writes are atomic via Lua. Put, Delete, and lease release each run as a single Redis Lua script so concurrent replicas can't interleave a partial index update.

  12. The on-disk layout is versioned by sentinel. A directory without .iiiris-cache-version containing hash-named files is treated as a pre-FS_CACHE2 v1 cache: the hash files are wiped (operator files preserved), and the sentinel is written with contents v2\n. Subsequent attaches are no-ops.

Migration

Per the "discard on upgrade" decision in the brief: upgrading from the pre-FS_CACHE2 single-process implementation wipes the existing hash entries on first attach. Cached renders re-populate on-demand. No tooling is shipped for in-place migration; operators needing a clean cut can stop iiirisd, rm -rf the cache root, and start the upgraded binary — the automatic path does the same thing.

No cacheKeyVersion bump is required (per scaffold.md cache-key versioning rules) — the cache keys are unchanged; only the on-disk index/coordination layout changed.

Test coverage

  • internal/cache/filesystem_test.go and filesystem_extra_test.go cover the local-mode public surface: Put/Get/Delete, eviction on overflow, Get-touches-LRU, startup trim, ignore-unrelated files, Stats, PurgeAll (preserves operator files), and Delete- missing-is-noop.
  • internal/cache/filesystem_migration_test.go covers the v1→v2 migration wipe (with operator-README preservation), fresh-directory init, and idempotent re-attach.
  • internal/cache/filesystem_concurrent_test.go spawns 4 worker subprocesses via os/exec re-entry against a shared cache root, verifying no corruption (all keys round-trip across worker boundaries) and that the cap is honored within walk- granularity tolerance after a final synchronous EvictNow. The same harness re-runs under CACHE_REDIS_ADDR to exercise distributed mode with a shared miniredis.
  • internal/cache/filesystem_redis_test.go covers the distributed- mode index mutations (Put/Get round-trip, overwrite delta, Delete index removal), Stats-from-Redis, eviction-under-cap, TryEvict skip-when-lease-held, reconciliation (ghost-drop + orphan-index), PurgeAll, and config rejection (backend=redis + coordinator.redis).
  • tools/cache-bench/ is the Phase 1 benchmark harness retained for future regression measurement (driving N concurrent worker processes against the no-index prototype with configurable workload mix and reporting Get/Put p50/p95/p99 plus standalone walk latency).
  • Multi-replica EFS is not covered in CI; release validation is the manual smoke playbook in ../deployment.md § Pre-release smoke: multi-replica EFS + Redis coordinator.

Out of scope

  • Embedded-DB coordinator (bbolt, SQLite) for local mode. Rejected during Phase 1 design: the workload (read-heavy, render-bounded write rate, single-host concurrency at most a handful of processes) doesn't justify the new dependency or the new on-disk format. The filesystem itself is a good enough index; the walk-on-evict cost (a sub-second event at 100k entries on Linux ext4) is paid only when actually evicting.
  • flock-coordinated distributed mode over NFS. Rejected: POSIX advisory locks over NFSv4 are "appears to work" with silent corruption under load. Redis is the only supported coordinator for shared-FS multi-replica.
  • Runtime detect-and-refuse for shared-FS-without-Redis misconfiguration (sentinel-file + heartbeat). Deferred from v1 in favor of documentation-only mitigation: a WARN log line on local-mode startup names the constraint (mode=local (single-host only)), and the operator docs are explicit. A runtime check can be added later without changing the contract; the absence of one is not a bug.
  • Online migration tooling to translate a pre-FS_CACHE2 cache to the v2 layout. The migration path wipes; the brief decided "discard on upgrade" because the local-mode v1 was always single-process-best-effort and a re-populate is cheaper to ship than a one-shot translator.
  • Cross-replica request coalescing. If two replicas simultaneously cache-miss the same render, both will render and the last writer wins (last os.Rename to the same hash path). The filesystem cache does not deduplicate the render work itself — that's a request-coalescing problem outside this spec.
  • Distributed-mode background eviction goroutine. Eviction fires only on the opportunistic Put-side trigger and on explicit EvictNow calls. A periodic-tick eviction goroutine would simplify cap-strictness reasoning but adds an always-on coordination cost; the Put-trigger model leaves the steady-state path lock-free.
  • An interface unifying *Filesystem and *FilesystemRedis beyond the cache-role interfaces. The two types deliberately duplicate the RenderCache surface rather than abstract behind a shared backend interface. The duplication is small (the per-role wrappers); the two implementations diverge enough internally (walk-on-evict vs Redis-index, flock vs lease) that an abstraction would either leak or be useless.