Cleanup pass over the three preceding commits.
Tabulate the cosine terms in the UI blurhash decoder instead of calling Math.cos
per pixel per component: 248us -> 75us for a 32x32 decode, and an album grid
mounts one decoder per tile. Output is unchanged, which the pinned pixel specs
enforce. The Go encoder already tabulated the same terms.
Drop the dead paths that deriving components inside Encode left behind: the
zero-size guard in components, the post-downscale empty check, and the
no-AC-factor branch, which cannot be reached now that the counts are always at
least 1x9. The empty-image check moves ahead of the derivation, where it belongs.
In the queue mock, look up item_artwork by its existing iaKey rather than
scanning the map, and hold the lock across EnqueueIfMissing through a shared
unlocked helper instead of releasing it mid-operation. Extract the duplicated
drain-and-resolve block in the scanner specs into one helper.
A full scan re-imports every track, so the per-entity artwork enqueue in
persistChanges fired for every album and artist in the library — measured as
100% of both on a live instance, and ~8.5k artists / ~16.7k Deezer fetches on a
96k-file library. Re-importing a track is no evidence the art changed: artist
art has no track-content source at all, and the albums re-derived to identical
hashes across three consecutive scans.
Enqueue through EnqueueIfMissing on a full scan, which anti-joins item_artwork
so only entities that never resolved are queued. Incremental scans keep the
unconditional Enqueue, where a re-import does mean the file changed. Doing this
at the enqueue site rather than skipping it wholesale keeps a first scan filling
in covers as it walks, instead of stalling every cover until the scan ends.
EnqueueMissing grows a priority argument and runs once at end of scan, so an
entity phase 1 never saw still resolves, at Scan priority rather than dropping
to Recheck behind the backfill.
album/artist/playlist/radio each carried the same eight lines, differing
only in element type and model.Kind. hydrateItems takes a ref callback
yielding an item's id and the ItemImage to fill, which is all that varied.
The len()==0 guards drop out: hydrateItemImages already short-circuits an
empty id list, and both loops are no-ops on an empty slice. applyItemImage
stays as-is; hydrateMediaFileArtwork and its own specs still use it.
Mechanical cleanups, no behavior change:
- 15 copies of the same id-extraction loop collapse to slice.Map (5 repo
mocks, the scanner's track sweep, 4 wantIDs assertions) and slice.ToMap
(6 index-by-id loops in the hydration specs).
- disc.go built a map[string]bool purely to dedup folder ids and then
walked it back into a slice; slice.Unique says that directly.
- folders_artist.go's image filter is slice.Filter over model.IsImageFile.
- mock_artwork_repo deleted from a map while ranging it; maps.DeleteFunc
states the intent.
- sort.Slice -> slices.SortFunc + cmp.Or; math.Min/Max -> builtin min/max;
make+copy -> bytes.Clone; strings.Split -> SplitSeq on a per-request
path; three-clause pixel loops -> for range.
- Reuse utils.BaseName where a stem was recomputed by hand. Not at
playlist_cover.go:27: that path is a full OS path and utils.BaseName
uses path.Base, which does not split backslashes.
- Drop a dead nil-guard in agents.go: getAgent returns a bare nil
interface, and a type assertion on nil already yields ok == false.
cmp.Or was rejected for the gate fallback (func types are not comparable,
does not compile) and for ItemArtwork.AttemptedAt (cmp.Or compares
time.Time with ==, which includes loc; IsZero does not).
Comments only; no executable code changed. Verified by comparing the Go
token stream of every touched file before and after: identical.
Removes 375 of the 1104 comment lines this branch added, targeting content
that belongs in a commit message or PR body rather than in the code:
rejected alternatives ("DeleteIfUnchanged, not Delete", "Waking all beats
routing by kind"), refactor history ("as the legacy reader did"), issue
references (#5798, #5597, #5376), benchmark numbers (~400ms, ~16k allocs),
and four persistence doc comments that duplicated the interface godoc in
model/artwork.go verbatim.
Comments predating this branch are left untouched.
The ASCII fixture trees in the e2e suites are deliberately kept above the
line budget: they diagram the fixture layout with its expected outcomes,
and every pre-existing block in those files carries one.
The enqueue sat in refreshCounters, which Put also reaches for an
ordinary metadata update, so renaming a playlist or editing its comment
re-resolved the cover. The 2x2 grid samples albums with random(), so
that silently handed the playlist a different cover for an edit that
touched no tracks -- and refetched remote artwork to do it.
It now happens where the track set actually changes: addTracks (which
Put-with-tracks and updatePlaylist both funnel through) and renumber
(reached from removeOrphans). Creation still enqueues even with no
tracks, since an imported m3u can carry an ExternalImageURL.
Reported by Codex on #5847.
Artwork state and its bytes outlive a deleted entity until the next
prune (@daily), and the serving path consulted only item_artwork, so a
Subsonic id or a signed public token kept serving a removed entity's
image in the meantime. Master's readers loaded the entity first, so this
was a regression.
The check goes in serveHash, the one path that can hand back a found
row's bytes: absent rows are already unavailable, and the provisional
and disc paths load their entity to resolve at all. Doing it there
instead of per-handler also settles who owns the invariant. Subsonic had
worked around it with artworkAccessible, whose comment described the
service "bypassing the library and private-playlist filters"; that
workaround is now deleted, since the service resolves through the
request-scoped repositories and enforces the filters itself.
Because those repositories are ctx-scoped, each caller says what it
wants by what it passes: Subsonic hands over the request context and so
gets visibility as well as existence, while the public image route
elevates like the Jellyfin one already did -- a token is the
authorization there, and a visibility check would hide a shared private
playlist, the very case shares exist for.
Two supporting fixes. albumRepository.Exists and mediaFileRepository
.Exists used the plain exists() helper, which applies no library filter,
so they reported rows in libraries the caller cannot see; they now count
through applyLibraryFilter as CountAll and artistRepository.Exists
already do. Neither had a production caller. RadioRepository gained the
Exists it lacked.
Tests follow the layers: the service refuses a found row whose entity is
gone, the repositories hide rows the caller may not see, and the
handlers only assert the context they hand over. Jellyfin needed no
change -- resolveArtworkID probes the entity tables, so a deleted item
yields an empty artwork id -- but that protection was incidental and
untested, so it is pinned now.
MarkFailed and Delete had no production caller: the worker only ever uses
MarkFailedIfUnchanged and DeleteIfUnchanged, which refuse to act on a row
a concurrent scan re-enqueued. Keeping the unconditional pair meant the
interface offered the racy variant under the more obvious name.
MarkFailedIfUnchanged does not build on MarkFailed, so nothing in the
implementation needed them either. The repository tests used them to put
a row into a backed-off state; they now do that directly.
The UI decoded every blurhash into a 32x32 bitmap and stretched it to fill its
container, so a non-square cover showed a full-box blur that collapsed into a
letterboxed image the moment it loaded. On the album detail page the
placeholder overhung the image by a third of the box height.
A blurhash string carries no aspect ratio of its own, so the dimensions have to
come from the server. artwork.width/height were already stored and read by
nobody; they now surface on ItemImage as imageWidth/imageHeight, hydrated
through the join that was already in place. Existing rows already carry them,
so no migration or rescan is needed.
A square request is padded rather than cropped, which aspect-fits the content
inside the square the server returns. That made the grid a second instance of
the same bug, so `square` now implies contain for the image as well as the
placeholder, instead of the two renderers reading it differently.
A first backfill enqueues artists before albums at a single priority, so
the drain took them in that order. Artists resolve through a
rate-limited agent, and gate() waits for its permit while holding a
worker slot, so the whole pool sat asleep in the limiter with every
album queued behind it.
Measured on a 96k-track library (29,115 artists to 6,949 albums, 4:1):
zero albums resolved in seven minutes, and roughly 3.3 hours before the
first album cover would have appeared. Splitting the drain gives each
class its own slots: albums now finish in under eight minutes while
artists trickle at the same 2/s they were always limited to.
The two budgets are carved out of MaxOpenConns so a second pool cannot
take connections the scanner and the UI need. Dequeue filters by kind,
and the drain index leads with item_kind so each pool seeks to its own
work instead of scanning past the other's backlog.
Hydration stamped the album's hash and blurhash onto an embedded-eligible
track that had no state row yet. Serving takes provisionalEmbedded for
exactly that case and returns the track's own embedded image, so the id
carried a content-version belonging to a different picture: every such
request fell back to no-cache instead of immutable, and a client keying
its cover cache on the blurhash paired the album's with the track's art.
AlbumCoverArtID had the mirror-image problem, building the album id from
the track's own ItemImage. It happened to work only because hydration
overwrote ImageHash in precisely the fallback cases; a track with its own
resolved art would have stamped that hash onto the album's id.
loadTracks and the playlist-track cursor were the only entity-page
paths that never hydrated artwork state, so a song reached through a
playlist behaved differently from the same song in the songs list:
Subsonic emitted a hashless coverArt id, which imghttp downgrades to
no-cache, and advertised art even for known-absent albums; Jellyfin
emitted AlbumPrimaryImageTag as the bare album id — a tag that never
changes when the cover does — and no blurhash at all.
The media-file hydration moves next to the other hydration helpers so
both paths share one implementation rather than growing a third.
The conflict clause updated only priority and retry_at, so a row that
already existed kept its original attempts and enqueued_at. The worker
measures the 12h give-up budget from enqueued_at, so any row that had
been pending across a longer gap — a server left off, an upgrade, a
laptop asleep — gave up on its very first attempt and settled absent.
The manual re-resolve endpoint is the sharpest case: it clears artwork
state and re-queues, but inherited the old row's spent window, so a
deliberate retry got one shot. A fresh request now gets a fresh budget,
with the mock updated to match.
Two gaps in when the pipeline re-resolves artwork:
The recheck job only requeued absent-state rows (hash=''), so an entity that was
never processed — added between scans, or on a server with the scanner disabled —
had no periodic safety net and stayed without artwork indefinitely. Add
EnqueueMissing(kind): a SQL set-difference enqueueing entities with no
item_artwork row at Recheck priority (ON CONFLICT DO NOTHING, so it never
disturbs a queued row). Run it once at startup and hourly alongside the
stale-absent recheck. Rename staleAbsentKinds -> recheckKinds accordingly.
Conversely, the config fingerprint included consts.Version, which embeds the git
SHA and so changed on every build, re-enqueueing every entity in the library
(~34k here) and re-querying external agents at the configured RPS for anything
without local art. Replace it with an explicit artworkEpoch constant, bumped
deliberately when resolution semantics change. The cases that motivated the
version input — absent art becoming available — are already covered by the
stale-absent and missing-row rechecks; only a corrected wrong-pick needs the
epoch. A test guards against reintroducing the version.
Entity-level artwork queries now take a typed model.Kind instead of a bare
prefix string. GetItemArtwork, DeleteForItem(s), GetInfoForItems,
EnqueueStaleAbsent, hydrateItemImages, enqueueBackfillKind and artwork.Refresh
convert to the prefix string only at the two real boundaries: the SQL
item_kind column (kind.Prefix() inside each repo) and external string inputs
(a new model.ParseKind for the nativeapi URL param, which also validates it).
The Backfill/stale-absent kind slices, the resolve.go dispatch switch, and the
kind→resource / kind→table lookup maps now use the Kind vars directly. The
queue lifecycle methods (MarkFailed/Delete*) keep string kinds — they operate
on a dequeued item's raw ItemKind column, which stays a string field, always
populated via kind.Prefix().
Removes every bare "al"/"ar"/… prefix literal from non-test code (27 -> 0);
behavior is unchanged.
Relocate the image-upload service from core to core/artwork as
artwork.Uploader, co-locating it with the resolver/worker/serving that own
the artwork state it invalidates. MaxImageUploadSize moves too — its only
callers are the two image-upload handlers — which lets core/image_upload.go
be deleted entirely.
Extract the shared "clear resolved state + re-queue at Bump" invalidation
into artwork.Refresh and fold nativeapi's refreshArtwork handler onto it,
removing the duplicated DeleteForItem+Enqueue block that had drifted into
three places.
The wire provider moves from core's set to artwork's; the
playlists.ImageUploadService binding moves to the top-level injector so
core/artwork stays unaware of playlists. Behavior is unchanged.
Cleanups surfaced by /simplify: the song mapper's own-art branch reimplemented
primaryImageTag's tag+blurhash-map construction (and its one-entry invariant) — route
it through the helper so that invariant lives in one place. Tie artworkChunkSize to a
whole multiple of artworkBatchSize so a cursor page re-chunks into even hydration
batches. Hoist a duplicated imageLoading && blurHash boolean in the album grid.
The fixture has title ties (e.g. three "Antenna" tracks), so the unscoped
positional comparison against GetAll only passed because SQLite's tie order
happened to coincide between the full scan and the pre-pass's id IN (...)
fetch. Scope it to onlySongs like the sibling ordering specs already do.
Real Jellyfin fills ImageTags from each item's own images before falling back
to the parent album, and Finamp checks imageTags.Primary before AlbumId. Our
mapper read only the album's image, so a track with distinct embedded art
silently showed the album cover.
Emit exactly one entry under ImageBlurHashes.Primary: Go marshals
map[string]string in sorted key order rather than insertion order, so a
second entry could pair the wrong blurhash with the image imageId resolves
to, and Finamp pins that pairing in its cache for 365 days.
Jellyfin's listSongs streamed media files via GetCursor, which never
hydrates artwork, so songs emitted entity-id image tags and no blurhash.
media_file now uses the same id pre-pass as the other three cursors
(album/artist/playlist), for consistency, but on a separate method,
GetCursorWithArtwork: GetCursor itself must stay untouched, since it's
also the scanner's hot path and the scanner never reads artwork.
Measured on 1,000,000 tracks, the pre-pass over all ids costs +41.8 MB
heap and +298 ms versus GetCursor's bounded +0.0 MB. The Jellyfin path
is paginated, though, so in practice it only ever pre-passes a page's
worth of ids, not the full library, and doesn't pay that cost.
The album, artist and playlist GetCursor built their own select and never
called hydrateArtwork, so every Jellyfin list endpoint (all six stream via
GetCursor) emitted entity-id image tags and no blurhash. Only GetAll
hydrated, which is why Subsonic and the native API were unaffected.
Each cursor now resolves its ordered/filtered/paginated id set with the
cheap id-only GetAllIDs query, then streams those ids in chunks through the
repo's existing GetAll, which already hydrates and applies the full select.
Max/Offset are consumed by the pre-pass alone; the chunk query carries only
the caller's filters, Sort and Order.
This also removes a pre-existing deep-pagination cost: keeping OFFSET out of
the joined query makes the pre-pass a covering index scan instead of paying
the library and annotation joins for every skipped row. Benchmarked on a
synthetic 100k-album DB with the real schema, page=500 at offset 90,000:
3.9ms via the id pre-pass, 52.5ms for the current shape, 192.7ms for a naive
join. An unpaginated full stream costs ~24% more, which is the trade.
GetAllIDs gains the annotation join whenever the caller's filters or sort
reference an annotation column (same gate CountAll uses), otherwise
Filters=IsFavorite and SortBy=PlayCount would fail in the pre-pass. The
playlist pre-pass repeats GetAll's columns so ORDER BY keeps resolving to
playlist.name rather than the joined user.name.
Drives hydrateArtwork itself (not applyItemImage directly) over tracks that
take each of the loop's continue branches, so a future edit moving the
AlbumImage fill below a continue would fail loudly instead of passing silently.
Two worker/enqueue fixes from review:
- playlistRepository.Put assigned the generated id to the caller's Playlist but passed
the stale copy (empty id) to refreshCounters, enqueueing a pl|"" row the worker
failed until the daily dangling purge while the real playlist went unresolved. Set
the id on the copy before enqueueing.
- The drain refresh batch only included found/foundStale, so a cover removed by a scan
(found -> absent) never notified clients, leaving the old immutable image displayed.
Broadcast absent outcomes too; precache still only warms found/foundStale.
The hydration fallback assigned a found album hash to every fallback track, but a
multi-disc track's CoverArtID emits a dc- id served from disc-specific art whose hash
is unknown at hydration time. Advertising dc-..._<albumHash> gave clients a content-
version that never changes when the disc image does, breaking id-based refresh. Only
stamp the album hash for single-disc tracks (DiscNumber == 0); multi-disc tracks stay
unhashed and rely on the correct ETag returned by the served response.
A generated-grid cover went stale after track mutations: nothing re-resolved the
playlist's artwork, and the request path deliberately never rebuilds the grid, so
serveEntity kept returning the old grid hash indefinitely. Enqueue pl artwork from
refreshCounters (the choke point for every track-set change); no clear, so the old
cover keeps serving until the worker rebuilds.
Round-1's hydration fix still copied the album's known-absent onto a non-eligible
(or own-absent) track, but MediaFile.CoverArtID routes a multi-disc track to disc
art, which resolves provisionally and is never known-absent. Marking it absent made
Subsonic omit coverArt so clients never requested a valid disc image. Only mark a
single-disc track absent, and only when its own art won't resolve.
An embedded-eligible track with no resolved item_artwork row inherited the album's
ImageAbsent, so when the album resolved absent (e.g. CoverArtPriority without
'embedded') the track's coverArt was omitted permanently — the client never
requested it, so the lazy mediafile path never resolved it — even though the
serving path would extract and serve the track's own embedded art. Hydration now
never copies the album's absence onto an eligible-but-unresolved track.
The provisional read-through and dangling re-enqueue used Enqueue, whose upsert
resets retry_at, so any browse of an unresolved entity that was backing off after
an external failure made it immediately eligible again — defeating the exponential
backoff during a provider outage. Add EnqueueBump, which raises priority but leaves
an existing row's retry_at intact, and route the serving path through it. Scan and
manual re-resolve keep Enqueue's reset (a detected change wants immediate retry).
Backfill enumerated every album, artist, playlist and radio via GetAll
and mapped out just the ID. GetAll materializes full entities (library
joins, participant/stats/tags JSON, annotation, artwork hydration), so on
a large library it loaded tens of thousands of heavy structs only to read
one field each — spiking transient RSS to ~1GB during the one-time
upgrade backfill, a memory risk on small NAS/Pi hardware.
Add GetAllIDs to the album, artist, playlist and radio repositories: it
reuses each repo's base row-set filter (library visibility, artist
content join, playlist userFilter) but projects only id, skipping the
heavy columns and post-processing. A per-repo parity test asserts
GetAllIDs returns exactly the same id set as GetAll.
Verified on a 727MB / 29k-artist production DB copy: peak RSS during
backfill dropped from ~1012MB to ~89MB, file descriptors flat, same
36,138 items enqueued.
Queue rows for deleted entities failed forever (Get -> ErrNotFound -> failed -> capped retries, unbounded). Add ArtworkQueueRepository.PurgeDangling, called from Prune next to the item_artwork purge. Separately, the found/absent path unconditionally deleted the dequeued row, erasing a concurrent scan re-enqueue; switch to DeleteIfUnchanged, which deletes only while retry_at still matches the dequeued value (verified retry_at is the column an Enqueue upsert resets).
Write falls through to a real write when the liveness touch fails, and sweep retention now matches the recorded mime's extension so obsolete variants are reclaimed.
Sweep now honors an mtime grace window (in-flight acquisitions and temp files), reacquired orphans reset the prune grace window, and the queue mock implements real stale-absent semantics.
