Regression from 04d5a556. Keying the resize cache on identity meant the
disc response no longer carried a content hash, and the full-size branch
set no ETag either — so WriteImageHeaders fell back to the empty hash
and emitted `ETag: ""` for every full-size disc image. Since ifNoneMatch
compares the unquoted value, a client echoing that back matched, and got
a 304 even after the image was replaced.
The full-size branch now carries the same identity validator the sized
branch uses, so it moves when the folder's images change.
WriteImageHeaders is hardened against the class as well: an empty
validator is no validator, so it is neither emitted nor matched.
Reported by Codex on #5847.
Third and last site in this class: findImageInFolder logged and skipped
an image the glob had already matched, so a permissions or mount failure
during the artist-folder traversal read as "no image here" and let
processItem settle the artist absent, discarding any artwork already
resolved.
A matched-but-unreadable file now propagates through fromArtistFolder
and lands as localError, the same as album folder art, embedded art and
uploads. A folder with no match stays a definitive miss.
Also normalizes the e2e path assertions with filepath.ToSlash: the
stored SourcePath is OS-native, so the forward-slash suffixes failed
all 23 folder specs on Windows.
Reported by Codex on #5847.
Media files are never enqueued — the scanner drops their state without
queueing them and the recheck kinds exclude them — so an unresolved
track keeps an empty ImageHash and SongToBaseItem always fell through to
AlbumPrimaryImageTag. Finamp then asks for the album image, nothing ever
requests mf-, and the read-through that resolves the track never fires:
the own-cover branch was unreachable for Jellyfin-only users.
An eligible, unresolved, not-known-absent track now advertises its id as
the Primary tag, which is what makes the client ask. The request serves
the embedded art and queues the track so the worker persists a real
hash. No blurhash is sent, since none exists yet and a fake would be
cached against that tag forever.
Serving gains the album fallback that made this safe to advertise: an
eligible track whose frame will not extract now falls back the way
CoverArtID does instead of answering with a placeholder.
Reported by Codex on #5847.
claim() reserves the whole batch before dispatch, but the cancellation
path returned without releasing what it had not yet started, leaving
those items in the in-flight set permanently — no later drain could
claim them again.
Harmless until the batch grew past the pool size; now a cancel strands
up to a full batch. The e2e harness cancels mid-drain after every
acquire, so it surfaced there first: one spec timed out waiting for an
item that had been claimed and abandoned, and the suite went from 59s
to 87s on CI.
The identity cache key used album.UpdatedAt alone, which a replaced
disc image does not necessarily move — the sized response would then
serve the old image indefinitely. The legacy reader folded ImportedAt
and the folder's ImagesUpdatedAt into its key for exactly this reason,
and loadAlbumFoldersPaths already returns that timestamp; the disc
reader was discarding it.
The resize-cache key was the content hash, which cannot be computed
without reading the file, so a warm cache never prevented the I/O: every
sized disc request read up to 20MB and hashed it before the lookup. The
legacy reader keyed on the id and the album's mtime and touched the file
only on a miss.
Disc art has no state row and therefore no stored hash, so the key is
that same identity — id, album mtime, DiscArtPriority — and the
selection chain now runs only when the cache misses. Full-size requests
stream the source instead of buffering and hashing it.
This matters more now that single-disc albums keep running the disc
chain, which puts every disc-tagged track without embedded art on this
path.
CI runs the suite under -race, which the local `make test` does not, so
this only showed up there: 230 specs passed and the detector still
failed the run.
The drain-pools spec started Run and never waited for it, so pool
goroutines outlived the spec and raced the config snapshot Ginkgo
restores on cleanup. It now cancels, unparks the blocked lookups and
waits for Run to return.
Two test doubles also had to become concurrency-safe, since the spec is
the first to resolve several artists at once: fakeImageAgent's call
counters, and MockAlbumRepo.GetAll, which records the last query options
on a read path. MockDataStore's lazy accessors get the same treatment —
only MediaFile was guarded before, and two pools now reach them
concurrently. ArtworkQueue takes an unlocked helper for its internal
Artwork call, since repoMu is not reentrant.
resolveLocalFile swallowed every os.Open error, so uploads, playlist
sidecars, a local M3U image and the artist image folder still had the
bug that was fixed for folder and embedded sources: a permission or
transient I/O error on a file that exists read as "no image here". The
worker then settled the item absent and dropped its queue row.
Uploads outrank every other source, so an unreadable one now stops the
chain rather than letting a lower-priority image be persisted in its
place. A genuinely missing file stays a clean miss.
Reported by Codex on #5847.
A first backfill enqueues artists before albums at a single priority, so
the drain took them in that order. Artists resolve through a
rate-limited agent, and gate() waits for its permit while holding a
worker slot, so the whole pool sat asleep in the limiter with every
album queued behind it.
Measured on a 96k-track library (29,115 artists to 6,949 albums, 4:1):
zero albums resolved in seven minutes, and roughly 3.3 hours before the
first album cover would have appeared. Splitting the drain gives each
class its own slots: albums now finish in under eight minutes while
artists trickle at the same 2/s they were always limited to.
The two budgets are carved out of MaxOpenConns so a second pool cannot
take connections the scanner and the UI need. Dequeue filters by kind,
and the drain index leads with item_kind so each pool seeks to its own
work instead of scanning past the other's backlog.
The pool was fed from a batch sized to the pool itself (2x concurrency)
with a WaitGroup barrier before the next dequeue, so one item burning
its external timeout idled every other slot until it finished. The
legacy cache warmer had no such barrier: it streamed through a pipeline
of 4.
Dequeuing well past the pool keeps the slots fed for the whole pass at
no extra cost, since DequeueBatch does not mark rows taken and was
already one query per pass. Acquiring a slot now also observes
cancellation, so a larger batch cannot delay shutdown.
Warming the resize cache re-read the two rows and the file the
acquisition had just written, so every acquired image cost two extra
queries and a second full read of a file whose bytes were still in
memory. processItem now hands back what it persisted and precache warms
from that, under the same cache key the serving path computes.
Resolving the admin user also moves behind the empty-queue check: it is
needed only to resolve private playlists, so an idle server no longer
runs a user lookup on every poll.
Sweep walked the whole store with no context, and RunPrune holds the
prune write lock for its full duration. In-flight acquisitions park on
the read lock, drain's WaitGroup never returns, and Run never reaches
its ctx.Err() check — so a SIGTERM during a daily prune over a large
store on slow storage waits out the container's grace period and dies
mid-remove.
GetOrPlaceholder folded model.ErrNotFound in with ErrUnavailable, so an
id matching no entity returned 200 and the placeholder PNG. That made
the ErrorDataNotFound branch in getCoverArt unreachable: Subsonic went
from error 70 to a successful placeholder, and Jellyfin's Primary image
endpoint from 404 to 200. Neither is ours to change.
An entity with no art and an id with no entity are different answers;
only the first is a placeholder.
Exhausting the 12h budget called writeAbsent unconditionally, so an
entity whose art was already resolved and serving lost it to a long
upstream outage: the hash went empty, clients fell back to the
placeholder, and the now-unreferenced bytes were freed by the next
prune even though nothing about the image had changed.
Exhaustion means the source stayed unreachable, not that the cover
disappeared, so absent is now recorded only when there is nothing to
keep.
Local sources swallowed their open errors, so a stale NFS/SMB mount was
indistinguishable from "this entity has no artwork": the chain returned
no reader, processItem took the absent branch, and the upsert replaced a
good content hash with the empty string. Clients then saw a placeholder
until the 1h request recheck or the 24h stale-absent sweep, and the
orphaned bytes became eligible for the next prune.
A candidate the resolver knows about — a file in the folder listing, a
track's own audio file — failing to open is not evidence of absence, so
it now forces a retry the same way an external agent error does.
The deleted artwork.go carried blank imports for image/gif and
x/image/webp. WebP came back via resize.go's gen2brain/webp, which
self-registers, but GIF did not: core/artwork claims GIF support in
mimeForFormat and extForMime while relying on an unrelated server
package to have imported the decoder.
The guard lives in the e2e suite because that test binary has no other
image/gif importer; a spec in core/artwork would pass regardless, since
animation_test.go imports the package non-blank.
ed4178a6 gated serveDisc on len(album.Discs) > 1, claiming parity with the legacy
reader. The legacy reader has no such gate: artwork.go dispatches every dc- id to
newDiscArtworkReader, whose Reader() walks DiscArtPriority unconditionally.
The gate also lost art. For a single-disc album whose only image is disc1.jpg,
the disc request skipped the chain and fell through to album art, which does not
match CoverArtPriority — so tracks tagged disc 1 (whose CoverArtID is a dc- id)
served nothing at all, where before they served disc1.jpg.
A single disc can legitimately have its own cover, distinct from the album's, and
DiscArtPriority is what expresses that preference. Drop the gate and restore the
single-disc e2e scenarios that covered it.
The serving cutover removed the album/disc/artist/mediafile/playlist/radio e2e
specs that documented the folder-selection rules and guarded the #5376/#5456/
#5451/#5457 regressions; nothing replaced them, so compareImageFiles and the
parent-fallback logic were left untested.
Restore them driving the real pipeline: a real scanner populates the folder
graph from an in-memory library, the real Worker drains the queue, and the real
Service serves. Folder-backed art is file-backed (served via os.Open, which the
in-memory FS can't satisfy) so its selection is asserted on the persisted state
row; store-backed and real-disk sources are asserted byte-for-byte. Single-disc
disc resolution now serves album art directly, so only multi-disc disc scenarios
are ported.
Two gaps in when the pipeline re-resolves artwork:
The recheck job only requeued absent-state rows (hash=''), so an entity that was
never processed — added between scans, or on a server with the scanner disabled —
had no periodic safety net and stayed without artwork indefinitely. Add
EnqueueMissing(kind): a SQL set-difference enqueueing entities with no
item_artwork row at Recheck priority (ON CONFLICT DO NOTHING, so it never
disturbs a queued row). Run it once at startup and hourly alongside the
stale-absent recheck. Rename staleAbsentKinds -> recheckKinds accordingly.
Conversely, the config fingerprint included consts.Version, which embeds the git
SHA and so changed on every build, re-enqueueing every entity in the library
(~34k here) and re-querying external agents at the configured RPS for anything
without local art. Replace it with an explicit artworkEpoch constant, bumped
deliberately when resolution semantics change. The cases that motivated the
version input — absent art becoming available — are already covered by the
stale-absent and missing-row rechecks; only a corrected wrong-pick needs the
epoch. A test guards against reintroducing the version.
An agent (notably Last.fm's album.getInfo) can advertise a cover URL that is
itself dead — a 404. sources.go's fromURL returned a generic error for any
non-200, so a dead URL was treated as a transient failure: it churned in
backoff and counted toward the circuit breaker, stalling valid lookups.
Map 404/410 to model.ErrNotFound in fromURL so a dead URL settles absent, and
collapse the near-identical fetchPlaylistImageURL (which already did this for
M3U covers) into it.
The worker's res.reader==nil && extError branch returned outcomeFailed with no
log, so a failing external cover lookup (agent error, dead image URL, download
timeout) was undiagnosable. Log the agent, entity, and underlying error at the
fetch site where it's in hand — this surfaced a Last.fm album.getInfo returning
an image URL that itself 404s.
Retry backoff now starts at 5s (was 15s) so a transient failure recovers on
essentially the next drain, and jitter widens to ±40% so a wave of correlated
failures doesn't re-clump into one poll.
Add a 12h give-up budget measured from enqueued_at: once the next backoff would
land past it, the worker stops retrying instead of grinding at a cap forever. A
bare failure settles absent (handed to the 24h stale-absent sweep, and still
recoverable on a page view); a found-stale keeps its already-served art. The
budget bounds the tail, so the separate 48h backoffCap is removed.
The exponential retry (base × 4^attempts, cap 48h) started at 5 minutes, so a
single transient failure — a timeout under load, an external blip — parked a
cover for 5 minutes even though a retry seconds later would have resolved it.
Start at 15s instead: transient failures recover almost immediately (15s → 1m
→ 4m → 16m …), while persistent failures still escalate to the 48h cap (now at
the 8th attempt instead of the 5th).
Entity-level artwork queries now take a typed model.Kind instead of a bare
prefix string. GetItemArtwork, DeleteForItem(s), GetInfoForItems,
EnqueueStaleAbsent, hydrateItemImages, enqueueBackfillKind and artwork.Refresh
convert to the prefix string only at the two real boundaries: the SQL
item_kind column (kind.Prefix() inside each repo) and external string inputs
(a new model.ParseKind for the nativeapi URL param, which also validates it).
The Backfill/stale-absent kind slices, the resolve.go dispatch switch, and the
kind→resource / kind→table lookup maps now use the Kind vars directly. The
queue lifecycle methods (MarkFailed/Delete*) keep string kinds — they operate
on a dequeued item's raw ItemKind column, which stays a string field, always
populated via kind.Prefix().
Removes every bare "al"/"ar"/… prefix literal from non-test code (27 -> 0);
behavior is unchanged.
Relocate the image-upload service from core to core/artwork as
artwork.Uploader, co-locating it with the resolver/worker/serving that own
the artwork state it invalidates. MaxImageUploadSize moves too — its only
callers are the two image-upload handlers — which lets core/image_upload.go
be deleted entirely.
Extract the shared "clear resolved state + re-queue at Bump" invalidation
into artwork.Refresh and fold nativeapi's refreshArtwork handler onto it,
removing the duplicated DeleteForItem+Enqueue block that had drifted into
three places.
The wire provider moves from core's set to artwork's; the
playlists.ImageUploadService binding moves to the top-level injector so
core/artwork stays unaware of playlists. Behavior is unchanged.
serveEntity now schedules a Bump recheck for an entity whose art was recorded
absent, so viewing a missing cover re-triggers resolution (e.g. after an
external source that was down during the scan comes back), matching the
request-time bump that already covers never-resolved entities.
Throttled by attempted_at against requestRecheckAge (1h) so repeatedly opening
a genuinely-absent page can't hammer external services. EnqueueBump preserves
an existing failed-state backoff via MAX(priority,...) and inserts a fresh,
immediately-eligible recheck for a settled-absent row (whose queue row was
already deleted).
- A negative size (Subsonic size / Jellyfin maxwidth accept signed ints) reached
resizeStaticImage, where the square path builds image.NewNRGBA(Rect(0,0,size,size))
— a giant rectangle that panics/OOMs. Clamp size<0 to 0 (full-size) at the Service
entry. Positive sizes were already clamped to the original.
- imghttp used a plain func Test with a table; convert to a Ginkgo DescribeTable with
the suite entry point in imghttp_suite_test.go (AGENTS.md test-framework requirement).
url.Parse puts the volume of file://C:/Music in Host, not Path, so localOSRoot dropped
it and returned /Music — os.Open/os.Stat then failed and folder/embedded art on Windows
looped as dangling. Rejoin the host volume, matching core/storage/local's newLocalStorage.
Two serving-correctness fixes from review:
- serveMediaFile served a persisted mf embedded image even after EnableMediaFileCoverArt
was turned off (the setting isn't in the config fingerprint, so found rows aren't
reprocessed). Direct mf- URLs now honor the setting at serve time and fall back to
disc/album art.
- The file-backed staleness check compared whole-second mtimes, so a same-second content
replacement (two writes in one second, or timestamp-preserving tools) could serve
different bytes under the old hash + immutable policy. RefMtime is now unix-nanoseconds
(no schema change; int64 column), detecting sub-second changes where the filesystem
records them.
Two worker/enqueue fixes from review:
- playlistRepository.Put assigned the generated id to the caller's Playlist but passed
the stale copy (empty id) to refreshCounters, enqueueing a pl|"" row the worker
failed until the daily dangling purge while the real playlist went unresolved. Set
the id on the copy before enqueueing.
- The drain refresh batch only included found/foundStale, so a cover removed by a scan
(found -> absent) never notified clients, leaving the old immutable image displayed.
Broadcast absent outcomes too; precache still only warms found/foundStale.
The ETag was the pixel hash of the original image, so a CoverArtQuality or
EnableWebPEncoding change altered the resized bytes without changing the ETag —
revalidating clients got a spurious 304 and kept the old encoding. Resized responses
now carry a representation ETag (hash + size + square + encode settings) used for the
ETag header and If-None-Match, while the immutable decision stays on the pixel hash
(URLs remain pixel-identity per the spec, so hash-suffixed clients keep zero-request
caching). Full-size originals fall back to the pixel hash as before.
serveEntity reads persisted item_artwork by id, bypassing the library and private-
playlist filters that the legacy entity-load applied. On the authenticated Subsonic
path a user could fetch artwork for an inaccessible album or someone else's private
playlist by guessing an id. getCoverArt now resolves the underlying entity through
the request-scoped (filtered) repositories and serves the placeholder when it is not
visible, so existence isn't leaked and the always-an-image invariant holds. The
public share (JWT-authorized) and Jellyfin (admin) paths are intentionally untouched.
Moving agent calls into the worker bypassed two behaviors of the aggregate provider:
Agents.GetArtistImages' guard for Unknown/Various Artists (a direct retriever call
could assign an unrelated image to a synthetic artist), and auxAlbum/auxArtist.Name's
DevPreserveUnicodeInExternalCalls normalization (records with typographic quotes/dashes
missed exact-name searches). Re-apply both before enumerating retrievers.
Deleting an artist/radio/playlist upload cleared the filename but left the found
item_artwork row and its hash, so lists kept advertising the deleted cover's
hash-suffixed immutable URL and clients could display it indefinitely. Call
EnqueueArtwork after the delete-side Put, symmetric with upload, so the state is
cleared and re-resolved to the next source (or absent).
DiscCoverArtID returns a dc- id for any track with DiscNumber>0, so serveDisc ran
the full DiscArtPriority chain even for single-disc albums, where a stray disc*/
embedded image could shadow higher-priority album art. Gate disc resolution on the
album having more than one disc, matching the legacy reader; single-disc tracks
serve album art directly.
A library configured with a file:// path stored absRoot as the raw URI, so Abs
produced strings like file:/music/cover.jpg that os.Open/os.Stat reject — folder,
upload and embedded art were treated as dangling on every request, looping forever.
Normalize a file:// path to its parsed OS path (the same root os.DirFS uses);
non-local schemes are left unchanged (out of scope, per the artwork-musicfs TODO).
A local ExternalImageURL was resolved through the external step and labelled
external, so placeBytes copied it into the content-addressed store and dropped its
path/mtime — replacing the file never tripped the staleness check. Classify local
references as file-backed (resolved in place, even on the request path) and keep
store-backed behaviour only for http(s) URLs.
SetImage cleared state and enqueued the bump before the caller stored the new
filename, so a worker drain in that window could resolve against the old (already
deleted) file and settle absent, leaving the upload unused until a later scan. Move
the invalidate+enqueue into EnqueueArtwork, which each caller now invokes after the
entity Put.
serveMediaFile delegated an absent/ineligible track straight to AlbumCoverArtID,
skipping the disc-specific lookup that MediaFile.CoverArtID (and the deleted legacy
reader) use. On multi-disc albums with per-disc images that served the album cover
instead of the configured disc artwork. Delegate through DiscCoverArtID.
bestImageURL selected the largest by size and only then parsed it, so a malformed
largest URL (e.g. a bad percent-escape) returned nil and shadowed a valid smaller
candidate, contradicting the documented skip-unparseable behavior. Parse per
candidate and compare sizes only among URLs that parse.
The provisional read-through and dangling re-enqueue used Enqueue, whose upsert
resets retry_at, so any browse of an unresolved entity that was backing off after
an external failure made it immediately eligible again — defeating the exponential
backoff during a provider outage. Add EnqueueBump, which raises priority but leaves
an existing row's retry_at intact, and route the serving path through it. Scan and
manual re-resolve keep Enqueue's reset (a detected change wants immediate retry).
