fix(artwork): re-resolve artwork when image files change on disk (#5965)

* fix(artwork): re-resolve artwork when image files change on disk

An image-only folder change (replaced, added, or deleted cover/artist
images, with no audio files touched) was detected by the scanner but never
reached the artwork queue, so clients kept seeing the old coverArt hash
until something else forced a re-resolution.

Phase 1 now diffs each changed folder's image list and imagesUpdatedAt
against the previously persisted folder row, and at the end of the phase
bulk-enqueues re-resolution for the affected entities: albums with tracks
in the folder or its direct children (covering disc subfolder layouts),
and, when an artist-pattern image is involved, artists with albums under
the folder's subtree, mirroring the artist resolver's upward search. The
artist mapping mirrors the resolver's sole-album-artist album selection.

New repository helpers keep the mapping set-based and light: folder
GetAllIDs, media_file GetAlbumIDsByFolder (distinct, indexed by
folder_id), and album GetSoleAlbumArtistIDs.

* refactor: simplify the image-change artwork enqueue after review

Load the previous folder image state through the existing GetFolderUpdateInfo
bulk pre-pass instead of a per-folder SELECT inside the persist transaction,
and skip the diff for new folders, whose artwork the scanner already enqueues
inline. Move the artist-image classification into core/artwork
(IsArtistImageFile) so the scanner shares the resolver's ArtistArtPriority
token grammar instead of re-parsing it (the copy mistreated image-folder as a
filename glob). Move the folder-subtree query into the folder repository
(GetSubtreeIDs) with LIKE escaping and expression-tree batching, share the
sole-album-artist predicate between the resolver and the album repository
(model.SoleAlbumArtistFilter), extract a chunked single-column query helper,
and deduplicate the ArtworkQueueItem literals behind scanArtworkItem.

* fix(persistence): keep slash-form paths in GetSubtreeIDs subtree predicates

The scanner hands GetSubtreeIDs io/fs slash-form paths, but filepath.Clean
rewrites them with backslashes on Windows while folder.path is stored with
forward slashes, so the descendant predicates matched nothing and nested
artist folders were never re-enqueued there. Normalize with path.Clean, like
HasAudioOutsideFolders does, and cover a nested path in the repo test.

* refactor(persistence): move the sole-album-artist rule into the album repository

SQLizer filters belong in the persistence package, not model. The rule
becomes an unexported filter shared by GetSoleAlbumArtistIDs and a new
GetBySoleAlbumArtist repository method, which the artist artwork resolver now
calls instead of building the squirrel filter itself.

* perf(scanner): resolve image-change artists in one query over album.folder_ids

The artist half of the image-change enqueue walked folder subtree IDs, then
media_file rows, then album rows, marshalling thousands of bound IDs through
the driver on each hop. Matching albums by their own folder_ids instead is one
statement, and folder_ids is the same source the artist resolver uses to
compute an artist's folders.

Benchmarked against a copy of the production DB (97k tracks, 10k folders,
7k albums): 87ms +/-196% -> 17.4ms +/-8%, 7.1MB -> 172KB, 103k -> 1.5k allocs.
The subtree predicate becomes a shared folderSubtreeFilter, so Folder
GetSubtreeIDs and Album GetSoleAlbumArtistIDs are no longer needed.

* fix(scanner): persist ancestor folders discovered by a quick scan

A quick scan skipped any new folder with no files of its own, so an artist
folder holding only album subfolders never got a row. Adding artist.jpg to it
later then produced no artwork enqueue: the entry was new, so the image diff
was skipped, and it has no tracks, so nothing was enqueued inline either.

Skip only genuinely empty new folders, matching what a full scan already
persists. This also fixes artist artwork resolving as absent for artists first
imported by a quick scan, since the resolver's folder climb needs that row.

Also normalizes the selective-scan preload paths with path.Clean, so its
descendant predicates match the stored slash-form paths on Windows.

* fix(persistence): chunk subtree paths and match artist globs by basename

Two regressions from earlier commits on this branch.

Collapsing the subtree query into a single statement dropped the chunking the
old GetSubtreeIDs had: each path expands into 3 OR terms and SQLite rejects an
expression tree deeper than 1000, measured at 166 paths. A library with more
artist-image folders than that (the prod copy has 158) would fail the whole
collect, dropping the album items with it, so the scanner now keeps them when
the artist query fails.

The artist-image classifier compared whole tokens after stripping album/, so a
directory-bearing glob like images/artist.* never matched the basenames the
scanner has. Match on path.Base, which is what album/artist.* already reduced
to; the resolver climbs parent folders, so an exact prefix is not knowable
here and a conservative match is the right failure direction.

* refactor(persistence): halve the repository surface this PR adds

Research on the four new repository methods found two were avoidable.

GetAlbumIDsByFolder now expands the changed folders to their direct children
in its own subquery, so Folder.GetAllIDs has no callers and is deleted, one
round trip per scan disappears, and the previously unchunked id/parent_id IN
lists are covered by the existing chunking.

GetBySoleAlbumArtist becomes an exported SoleAlbumArtistFilter, matching the
ParticipantIDFilter precedent for sharing a Sqlizer with core/, so the rule
still lives in persistence but AlbumRepository gains nothing and the mock shim
that ignored the artist filter is gone.

Also drops queryAllSliceChunked, now callerless, in favour of the file-local
slices.Chunk convention used by the sibling folder queries.

Rejected on measurement: matching the album path by album.folder_ids is exactly
equivalent (13975 pairs, zero difference) but has no index, so it scans every
album and runs 5-200x slower than the media_file route.

* refactor(persistence): stop reading the deprecated album_artist_id column

Both artist lookups this PR touches now go through participation, matching
the precedent in core/archiver.go and share_repository.go.

SoleAlbumArtistFilter uses ParticipantIDFilter, which is also faster: the
album_artists unique constraint is a covering index for it, while the old
column needed album_artist_album_id plus a row fetch.

GetSoleAlbumArtistIDsInSubtrees reads the sole artist out of the participants
JSON it already parses for the sole-artist check, rather than joining back to
album_artists, which measured ~1.6x slower on a prod-sized copy.

Verified equivalent on that copy: 6828 sole-artist albums and 1088 subtree
artists resolve identically via the column, the join and the JSON. The tests
now set a deliberately wrong album_artist_id so they fail if either query
starts reading it again.

* docs: trim comments that carry rationale belonging in commit messages

Five comments had grown past the budget with benchmark numbers, rejected
alternatives, and a duplicate of the constant's own explanation.

* refactor(scanner): move the image-change enqueue into phase_1_folders

The three functions were methods on phaseFolders, so they belong with the
type; phase_1_image_changes.go also read like a fifth phase, which it wasn't.

* refactor(scanner): extract the image-change collector into its own type

phaseFolders no longer owns the per-library map and the mapping methods; it
records into a collector and asks it to enqueue once. The collector keeps the
library alongside the folders, so enqueue needs only ctx and the datastore.

* refactor(scanner): simplify enqueue method by removing redundant datastore parameter

Signed-off-by: Deluan <deluan@navidrome.org>

* docs(scanner): drop the stale zero-value claim on imageChangeCollector

The collector now takes its datastore at construction, so the zero value is
no longer usable.

* fix(scanner): pin the persist stage to concurrency 1 and guard the collector

The stage relied on go-pipeline defaulting to one worker; stating it at the
stage makes the constraint visible where someone would change it. The
collector takes a mutex too, so the type is safe on its own terms rather than
by configuration.

---------

Signed-off-by: Deluan <deluan@navidrome.org>
This commit is contained in:
Deluan Quintão 2026-08-16 13:24:07 -04:00 committed by GitHub
parent 42dfdf49da
commit 5b758fc20c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
16 changed files with 602 additions and 51 deletions

View File

@ -9,13 +9,14 @@ import (
"io/fs"
"net/url"
"os"
"path"
"strings"
"github.com/Masterminds/squirrel"
"github.com/navidrome/navidrome/conf"
"github.com/navidrome/navidrome/core/agents"
"github.com/navidrome/navidrome/core/ffmpeg"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/persistence"
)
// resolution is one attempted acquisition outcome for an entity.
@ -248,12 +249,7 @@ func (r *resolver) resolveArtist(ctx context.Context, artistID string) (resoluti
}
// Only consider albums where the artist is the sole album artist.
als, err := r.ds.Album(ctx).GetAll(model.QueryOptions{
Filters: squirrel.And{
squirrel.Eq{"album_artist_id": artistID},
squirrel.Eq{"json_array_length(participants, '$.albumartist')": 1},
},
})
als, err := r.ds.Album(ctx).GetAll(model.QueryOptions{Filters: persistence.SoleAlbumArtistFilter(artistID)})
if err != nil {
return resolution{}, err
}
@ -525,6 +521,22 @@ func resolveFolderFile(ctx context.Context, lib libraryView, imgFiles []string,
return resolveFolderSource(lib, fromExternalFile(ctx, lib.FS, imgFiles, pattern))
}
// IsArtistImageFile reports whether a file name matches any file-glob token of ArtistArtPriority.
// Basename-only on purpose: the chain climbs parent folders, so a token's prefix is not fixed.
func IsArtistImageFile(name string) bool {
name = strings.ToLower(name)
for pattern := range strings.SplitSeq(strings.ToLower(conf.Server.ArtistArtPriority), ",") {
pattern = strings.TrimSpace(pattern)
if pattern == "" || pattern == externalCandidate || pattern == "image-folder" {
continue
}
if ok, _ := path.Match(path.Base(pattern), name); ok {
return true
}
}
return false
}
func resolveArtistImageFolder(ar *model.Artist) (resolution, bool) {
folder := conf.Server.ArtistImageFolder
if folder == "" {

View File

@ -21,6 +21,31 @@ import (
. "github.com/onsi/gomega"
)
var _ = Describe("IsArtistImageFile", func() {
BeforeEach(func() {
DeferCleanup(configtest.SetupConfig())
})
It("matches bare and album/-prefixed glob tokens, case-insensitively", func() {
conf.Server.ArtistArtPriority = "artist.*, album/artistfolder.*, external"
Expect(IsArtistImageFile("Artist.jpg")).To(BeTrue())
Expect(IsArtistImageFile("artistfolder.png")).To(BeTrue())
Expect(IsArtistImageFile("cover.jpg")).To(BeFalse())
})
It("matches a directory-bearing glob by its basename", func() {
conf.Server.ArtistArtPriority = "images/artist.*, external"
Expect(IsArtistImageFile("artist.jpg")).To(BeTrue())
Expect(IsArtistImageFile("cover.jpg")).To(BeFalse())
})
It("does not treat non-file tokens as globs", func() {
conf.Server.ArtistArtPriority = "image-folder, external"
Expect(IsArtistImageFile("image-folder")).To(BeFalse())
Expect(IsArtistImageFile("external")).To(BeFalse())
})
})
var _ = Describe("resolveItem", func() {
var (
ctx context.Context

View File

@ -144,6 +144,9 @@ type AlbumRepository interface {
Get(id string) (*Album, error)
GetAll(...QueryOptions) (Albums, error)
GetAllIDs(...QueryOptions) ([]string, error)
// GetSoleAlbumArtistIDsInSubtrees returns the sole album artists of the albums with folders in
// any of the given library-relative subtrees.
GetSoleAlbumArtistIDsInSubtrees(lib Library, paths ...string) ([]string, error)
GetCursor(...QueryOptions) (AlbumCursor, error)
GetYears(libraryIDs ...int) ([]int, error)

View File

@ -76,8 +76,10 @@ func NewFolder(lib Library, folderPath string) *Folder {
type FolderCursor iter.Seq2[Folder, error]
type FolderUpdateInfo struct {
UpdatedAt time.Time
Hash string
UpdatedAt time.Time
Hash string
ImageFiles []string
ImagesUpdatedAt time.Time
}
type FolderRepository interface {

View File

@ -555,6 +555,9 @@ type MediaFileRepository interface {
GetCursor(options ...QueryOptions) (MediaFileCursor, error)
// GetAllIDs returns just the media_file IDs for the same row set as GetAll.
GetAllIDs(options ...QueryOptions) ([]string, error)
// GetAlbumIDsByFolder returns the distinct IDs of albums with non-missing tracks in the given
// folders or their direct children.
GetAlbumIDsByFolder(lib Library, folderIDs ...string) ([]string, error)
// GetCursorWithArtwork streams like GetCursor, hydrated, so callers that render images don't
// pay the scanner's per-row cost; it uses the same id pre-pass as the other cursors.
GetCursorWithArtwork(options ...QueryOptions) (MediaFileCursor, error)

View File

@ -270,6 +270,39 @@ func (r *albumRepository) GetAllIDs(options ...model.QueryOptions) ([]string, er
return ids, err
}
// soleAlbumArtistFilter matches albums with exactly one album artist. The artist artwork
// resolver and the scanner's image-change enqueue must select the same albums.
var soleAlbumArtistFilter = Eq{"json_array_length(participants, '$.albumartist')": 1}
// SoleAlbumArtistFilter matches the albums where the given artist is the only album artist.
// Matches by album-artist participation, not the deprecated album_artist_id column.
func SoleAlbumArtistFilter(artistID string) Sqlizer {
return And{ParticipantIDFilter("album", artistID, model.RoleAlbumArtist), soleAlbumArtistFilter}
}
// GetSoleAlbumArtistIDsInSubtrees matches albums by their own folder_ids, which is the resolver's
// notion of an album's folders.
func (r *albumRepository) GetSoleAlbumArtistIDsInSubtrees(lib model.Library, paths ...string) ([]string, error) {
if len(paths) == 0 {
return nil, nil
}
ids := []string{}
// Repeated IDs across chunks are fine: the queue upserts by PK.
for chunk := range slices.Chunk(paths, subtreePathChunkSize) {
inSubtree := Exists("json_each(album.folder_ids) je join folder on folder.id = je.value",
folderSubtreeFilter(lib, chunk))
// Sole album artist, so participants[0] is the only one.
sq := Select("distinct json_extract(participants, '$.albumartist[0].id')").From("album").
Where(And{soleAlbumArtistFilter, inSubtree})
var chunkIDs []string
if err := r.queryAllSlice(sq, &chunkIDs); err != nil {
return nil, err
}
ids = append(ids, chunkIDs...)
}
return ids, nil
}
func (r *albumRepository) GetCursor(options ...model.QueryOptions) (model.AlbumCursor, error) {
ids, err := r.GetAllIDs(options...)
if err != nil {

View File

@ -1,6 +1,7 @@
package persistence
import (
"context"
"errors"
"fmt"
"sort"
@ -20,9 +21,10 @@ import (
var _ = Describe("AlbumRepository", func() {
var albumRepo *albumRepository
var ctx context.Context
BeforeEach(func() {
ctx := request.WithUser(GinkgoT().Context(), model.User{ID: "userid", UserName: "johndoe"})
ctx = request.WithUser(GinkgoT().Context(), model.User{ID: "userid", UserName: "johndoe"})
albumRepo = NewAlbumRepository(ctx, GetDBXBuilder()).(*albumRepository)
})
@ -96,6 +98,77 @@ var _ = Describe("AlbumRepository", func() {
})
})
Describe("GetSoleAlbumArtistIDsInSubtrees", func() {
It("returns the sole album artists of albums with folders in the subtree", func() {
folderRepo := newFolderRepository(ctx, GetDBXBuilder())
lib, err := NewLibraryRepository(ctx, GetDBXBuilder()).Get(1)
Expect(err).ToNot(HaveOccurred())
inTree := model.NewFolder(*lib, "SubtreeAlbums/Artist")
outTree := model.NewFolder(*lib, "OtherTree/Artist")
Expect(folderRepo.Put(inTree)).To(Succeed())
Expect(folderRepo.Put(outTree)).To(Succeed())
// album_artist_id is deliberately wrong: the artist must come from participants
inAl := model.Album{ID: "subtree-in-al", Name: "In", LibraryID: 1, AlbumArtistID: "999", FolderIDs: []string{inTree.ID},
Participants: model.Participants{model.RoleAlbumArtist: []model.Participant{{Artist: artistKraftwerk}}}}
outAl := model.Album{ID: "subtree-out-al", Name: "Out", LibraryID: 1, AlbumArtistID: "3", FolderIDs: []string{outTree.ID},
Participants: model.Participants{model.RoleAlbumArtist: []model.Participant{{Artist: artistBeatles}}}}
duoAl := model.Album{ID: "subtree-duo-al", Name: "Duo", LibraryID: 1, AlbumArtistID: "5", FolderIDs: []string{inTree.ID},
Participants: model.Participants{model.RoleAlbumArtist: []model.Participant{{Artist: artistPunctuation}, {Artist: artistBeatles}}}}
for _, al := range []model.Album{inAl, outAl, duoAl} {
Expect(albumRepo.Put(&al)).To(Succeed())
}
DeferCleanup(func() {
_, _ = GetDBXBuilder().NewQuery("DELETE FROM album WHERE id LIKE 'subtree-%'").Execute()
_, _ = GetDBXBuilder().NewQuery("DELETE FROM folder WHERE path LIKE 'SubtreeAlbums%' OR path LIKE 'OtherTree%'").Execute()
})
ids, err := albumRepo.GetSoleAlbumArtistIDsInSubtrees(*lib, "SubtreeAlbums")
Expect(err).ToNot(HaveOccurred())
Expect(ids).To(ConsistOf("2")) // sole artist in the subtree; the duo and the outside album are excluded
})
It("stays under SQLite's expression tree depth limit with many paths", func() {
lib, err := NewLibraryRepository(ctx, GetDBXBuilder()).Get(1)
Expect(err).ToNot(HaveOccurred())
paths := make([]string, 200)
for i := range paths {
paths[i] = fmt.Sprintf("DepthProbe/Folder%d", i)
}
_, err = albumRepo.GetSoleAlbumArtistIDsInSubtrees(*lib, paths...)
Expect(err).ToNot(HaveOccurred())
})
It("returns nothing when given no paths", func() {
lib, err := NewLibraryRepository(ctx, GetDBXBuilder()).Get(1)
Expect(err).ToNot(HaveOccurred())
ids, err := albumRepo.GetSoleAlbumArtistIDsInSubtrees(*lib)
Expect(err).ToNot(HaveOccurred())
Expect(ids).To(BeEmpty())
})
})
Describe("SoleAlbumArtistFilter", func() {
It("matches only albums where the artist is the sole album artist", func() {
// album_artist_id is deliberately wrong: matching must come from participation
sole := model.Album{ID: "sole-artist-al", Name: "Sole", LibraryID: 1, AlbumArtistID: "999",
Participants: model.Participants{model.RoleAlbumArtist: []model.Participant{{Artist: artistKraftwerk}}}}
duo := model.Album{ID: "duo-artist-al", Name: "Duo", LibraryID: 1, AlbumArtistID: "999",
Participants: model.Participants{model.RoleAlbumArtist: []model.Participant{{Artist: artistKraftwerk}, {Artist: artistBeatles}}}}
Expect(albumRepo.Put(&sole)).To(Succeed())
Expect(albumRepo.Put(&duo)).To(Succeed())
DeferCleanup(func() {
_, _ = GetDBXBuilder().NewQuery("DELETE FROM album WHERE id IN ('sole-artist-al', 'duo-artist-al')").Execute()
})
als, err := albumRepo.GetAll(model.QueryOptions{Filters: SoleAlbumArtistFilter("2")})
Expect(err).ToNot(HaveOccurred())
Expect(als).To(HaveLen(1))
Expect(als[0].ID).To(Equal(sole.ID))
})
})
Describe("GetAll", func() {
var GetAll = func(opts ...model.QueryOptions) (model.Albums, error) {
albums, err := albumRepo.GetAll(opts...)

View File

@ -6,9 +6,7 @@ import (
"fmt"
"iter"
"maps"
"os"
"path"
"path/filepath"
"slices"
"strings"
"time"
@ -147,9 +145,8 @@ func (r folderRepository) getFolderUpdateInfoBatch(lib model.Library, targetPath
pathConditions := make(Or, 0, len(targetPaths)*2)
for _, targetPath := range targetPaths {
// Clean the path to normalize it. Paths stored in the folder table do not have leading/trailing slashes.
cleanPath := strings.TrimPrefix(targetPath, string(os.PathSeparator))
cleanPath = filepath.Clean(cleanPath)
// Slash-form like the stored paths; filepath.Clean would backslash them on Windows.
cleanPath := path.Clean(strings.TrimPrefix(targetPath, "/"))
// Include the target folder itself by ID
folderIDs = append(folderIDs, model.FolderID(lib, cleanPath))
@ -172,11 +169,13 @@ func (r folderRepository) getFolderUpdateInfoBatch(lib model.Library, targetPath
// queryFolderUpdateInfo executes the query and returns the result map
func (r folderRepository) queryFolderUpdateInfo(where And) (map[string]model.FolderUpdateInfo, error) {
sq := r.newSelect().Columns("id", "updated_at", "hash").Where(where)
sq := r.newSelect().Columns("id", "updated_at", "hash", "image_files", "images_updated_at").Where(where)
var res []struct {
ID string
UpdatedAt time.Time
Hash string
ID string
UpdatedAt time.Time
Hash string
ImageFiles string
ImagesUpdatedAt time.Time
}
err := r.queryAll(sq, &res)
if err != nil {
@ -184,11 +183,41 @@ func (r folderRepository) queryFolderUpdateInfo(where And) (map[string]model.Fol
}
m := make(map[string]model.FolderUpdateInfo, len(res))
for _, f := range res {
m[f.ID] = model.FolderUpdateInfo{UpdatedAt: f.UpdatedAt, Hash: f.Hash}
info := model.FolderUpdateInfo{UpdatedAt: f.UpdatedAt, Hash: f.Hash, ImagesUpdatedAt: f.ImagesUpdatedAt}
if f.ImageFiles != "" {
if err := json.Unmarshal([]byte(f.ImageFiles), &info.ImageFiles); err != nil {
return nil, fmt.Errorf("parsing folder image_files: %w", err)
}
}
m[f.ID] = info
}
return m, nil
}
// subtreePathChunkSize bounds how many paths one folderSubtreeFilter may expand into: each adds
// 3 OR terms, and SQLite rejects an expression tree deeper than 1000 (measured: 166 paths).
const subtreePathChunkSize = 100
// folderSubtreeFilter matches the folders at the given library-relative paths and all their
// descendants. A path of "" or "." selects the whole library, so it drops the path conditions.
func folderSubtreeFilter(lib model.Library, paths []string) Sqlizer {
conds := make(Or, 0, len(paths)*3)
for _, p := range paths {
// Paths are io/fs slash-form; filepath.Clean would backslash them on Windows.
cleanPath := path.Clean(strings.TrimPrefix(p, "/"))
if cleanPath == "." {
return And{Eq{"folder.library_id": lib.ID}, Eq{"folder.missing": false}}
}
conds = append(conds,
Eq{"folder.id": model.FolderID(lib, cleanPath)},
// Direct children have path = cleanPath; deeper descendants match the prefix
Eq{"folder.path": cleanPath},
Expr(`folder.path LIKE ? ESCAPE '\'`, escapeLikePrefix(cleanPath)+"/%"),
)
}
return And{Eq{"folder.library_id": lib.ID}, Eq{"folder.missing": false}, conds}
}
// HasAudioOutsideFolders reports whether any folder in parent's subtree
// (including parent itself) contains audio files and is not one of the given
// folder IDs. LIKE wildcards in the parent path are escaped, so it is always

View File

@ -9,6 +9,7 @@ import (
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/model/request"
"github.com/navidrome/navidrome/tests"
"github.com/navidrome/navidrome/utils/slice"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/pocketbase/dbx"
@ -43,6 +44,42 @@ var _ = Describe("FolderRepository", func() {
_, _ = conn.NewQuery(fmt.Sprintf("DELETE FROM library WHERE id = %d", otherLib.ID)).Execute()
})
Describe("folderSubtreeFilter", func() {
var parent, child, grandchild, other *model.Folder
matching := func(paths ...string) []string {
GinkgoHelper()
folders, err := repo.GetAll(model.QueryOptions{Filters: folderSubtreeFilter(testLib, paths)})
Expect(err).ToNot(HaveOccurred())
return slice.Map(folders, func(f model.Folder) string { return f.ID })
}
BeforeEach(func() {
parent = model.NewFolder(testLib, "TestSubtree")
child = model.NewFolder(testLib, "TestSubtree/Child")
grandchild = model.NewFolder(testLib, "TestSubtree/Child/Grandchild")
other = model.NewFolder(testLib, "TestSubtreeOther")
for _, f := range []*model.Folder{parent, child, grandchild, other} {
Expect(repo.Put(f)).To(Succeed())
}
DeferCleanup(func() {
_, _ = conn.NewQuery("DELETE FROM folder WHERE name LIKE 'TestSubtree%' OR path LIKE 'TestSubtree%'").Execute()
})
})
It("matches a folder and all its descendants", func() {
Expect(matching("TestSubtree")).To(ConsistOf(parent.ID, child.ID, grandchild.ID))
})
It("matches the descendants of a nested slash-form path", func() {
Expect(matching("TestSubtree/Child")).To(ConsistOf(child.ID, grandchild.ID))
})
It("matches the whole library for the root path", func() {
Expect(matching(".")).To(ContainElements(parent.ID, child.ID, grandchild.ID, other.ID))
})
})
Describe("GetFolderUpdateInfo", func() {
Context("with no target paths", func() {
It("returns all folders in the library", func() {

View File

@ -318,6 +318,26 @@ func (r *mediaFileRepository) GetAllIDs(options ...model.QueryOptions) ([]string
return ids, err
}
func (r *mediaFileRepository) GetAlbumIDsByFolder(lib model.Library, folderIDs ...string) ([]string, error) {
ids := []string{}
for chunk := range slices.Chunk(folderIDs, 200) {
// A folder's own cover also covers albums whose tracks sit in its disc subfolders.
inFolders := Select("f.id").From("folder f").Where(And{
Eq{"f.library_id": lib.ID},
Eq{"f.missing": false},
Or{Eq{"f.id": chunk}, Eq{"f.parent_id": chunk}},
})
sq := Select("distinct album_id").From("media_file").
Where(And{Eq{"missing": false}, ConcatExpr("folder_id IN (", inFolders, ")")})
var chunkIDs []string
if err := r.queryAllSlice(sq, &chunkIDs); err != nil {
return nil, err
}
ids = append(ids, chunkIDs...)
}
return ids, nil
}
// GetCursorWithArtwork streams the same rows as GetCursor, hydrated, via an id pre-pass.
func (r *mediaFileRepository) GetCursorWithArtwork(options ...model.QueryOptions) (model.MediaFileCursor, error) {
ids, err := r.GetAllIDs(options...)

View File

@ -30,6 +30,54 @@ var _ = Describe("MediaRepository", func() {
mr = NewMediaFileRepository(ctx, GetDBXBuilder())
})
Describe("GetAlbumIDsByFolder", func() {
var lib model.Library
var albumRoot, disc1, sibling *model.Folder
BeforeEach(func() {
ctx := request.WithUser(log.NewContext(context.TODO()), model.User{ID: "userid"})
libPtr, err := NewLibraryRepository(ctx, GetDBXBuilder()).Get(1)
Expect(err).ToNot(HaveOccurred())
lib = *libPtr
folderRepo := newFolderRepository(ctx, GetDBXBuilder())
albumRoot = model.NewFolder(lib, "ByFolder/Album")
disc1 = model.NewFolder(lib, "ByFolder/Album/CD1")
sibling = model.NewFolder(lib, "ByFolder/Other")
for _, f := range []*model.Folder{albumRoot, disc1, sibling} {
Expect(folderRepo.Put(f)).To(Succeed())
}
// Tracks live in the disc subfolder; the sibling album is the negative control.
Expect(mr.Put(&model.MediaFile{ID: "fol-mf-1", LibraryID: 1, AlbumID: "fol-al-1", FolderID: disc1.ID, Path: "t/1.mp3"})).To(Succeed())
Expect(mr.Put(&model.MediaFile{ID: "fol-mf-2", LibraryID: 1, AlbumID: "fol-al-1", FolderID: disc1.ID, Path: "t/2.mp3"})).To(Succeed())
Expect(mr.Put(&model.MediaFile{ID: "fol-mf-3", LibraryID: 1, AlbumID: "fol-al-2", FolderID: sibling.ID, Path: "t/3.mp3"})).To(Succeed())
Expect(mr.Put(&model.MediaFile{ID: "fol-mf-4", LibraryID: 1, AlbumID: "fol-al-3", FolderID: disc1.ID, Path: "t/4.mp3", Missing: true})).To(Succeed())
DeferCleanup(func() {
_, _ = GetDBXBuilder().NewQuery("DELETE FROM media_file WHERE id LIKE 'fol-mf-%'").Execute()
_, _ = GetDBXBuilder().NewQuery("DELETE FROM folder WHERE path LIKE 'ByFolder%' OR name = 'ByFolder'").Execute()
})
})
It("returns the distinct album IDs of non-missing tracks in the folder", func() {
ids, err := mr.GetAlbumIDsByFolder(lib, disc1.ID)
Expect(err).ToNot(HaveOccurred())
Expect(ids).To(ConsistOf("fol-al-1"))
})
It("also matches albums whose tracks are in a direct child of the folder", func() {
// A cover in the album root must reach the album whose tracks sit in CD1
ids, err := mr.GetAlbumIDsByFolder(lib, albumRoot.ID)
Expect(err).ToNot(HaveOccurred())
Expect(ids).To(ConsistOf("fol-al-1"))
})
It("does not match albums outside the folder", func() {
ids, err := mr.GetAlbumIDsByFolder(lib, albumRoot.ID)
Expect(err).ToNot(HaveOccurred())
Expect(ids).ToNot(ContainElement("fol-al-2"))
})
})
Describe("GetCursor", func() {
It("yields the same media files as GetAll", func() {
opts := model.QueryOptions{Sort: "title"}

View File

@ -10,21 +10,24 @@ import (
"slices"
"time"
"github.com/navidrome/navidrome/core/artwork"
"github.com/navidrome/navidrome/core/playlists"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/utils/chrono"
)
func newFolderEntry(job *scanJob, id, path string, updTime time.Time, hash string) *folderEntry {
func newFolderEntry(job *scanJob, id, path string, info model.FolderUpdateInfo) *folderEntry {
f := &folderEntry{
id: id,
job: job,
path: path,
audioFiles: make(map[string]fs.DirEntry),
imageFiles: make(map[string]fs.DirEntry),
albumIDMap: make(map[string]string),
updTime: updTime,
prevHash: hash,
id: id,
job: job,
path: path,
audioFiles: make(map[string]fs.DirEntry),
imageFiles: make(map[string]fs.DirEntry),
albumIDMap: make(map[string]string),
updTime: info.UpdatedAt,
prevHash: info.Hash,
prevImageFiles: info.ImageFiles,
prevImagesUpdatedAt: info.ImagesUpdatedAt,
}
return f
}
@ -42,12 +45,15 @@ type folderEntry struct {
numSubFolders int
imagesUpdatedAt time.Time
prevHash string // Previous hash from DB
tracks model.MediaFiles
albums model.Albums
albumIDMap map[string]string
artists model.Artists
tags model.TagList
missingTracks []*model.MediaFile
// Previous image state from DB, to detect image-only changes
prevImageFiles []string
prevImagesUpdatedAt time.Time
tracks model.MediaFiles
albums model.Albums
albumIDMap map[string]string
artists model.Artists
tags model.TagList
missingTracks []*model.MediaFile
}
func (f *folderEntry) hasNoFiles() bool {
@ -69,6 +75,21 @@ func (f *folderEntry) isOutdated() bool {
return f.prevHash != f.hash()
}
// imagesChanged reports whether the folder's image files differ from the previously persisted
// state, and whether an artist image is involved (present in the old or the new list).
func (f *folderEntry) imagesChanged() (changed, artistImage bool) {
newNames := slices.Sorted(maps.Keys(f.imageFiles))
prevNames := slices.Sorted(slices.Values(f.prevImageFiles))
// Both empty also skips the timestamp check, which is noise for image-less folders.
if len(prevNames) == 0 && len(newNames) == 0 {
return false, false
}
if slices.Equal(prevNames, newNames) && f.prevImagesUpdatedAt.Equal(f.imagesUpdatedAt) {
return false, false
}
return true, slices.ContainsFunc(slices.Concat(prevNames, newNames), artwork.IsArtistImageFile)
}
func (f *folderEntry) toFolder() *model.Folder {
folder := model.NewFolder(f.job.lib, f.path)
folder.NumAudioFiles = len(f.audioFiles)

View File

@ -41,7 +41,7 @@ var _ = Describe("folder_entry", func() {
Hash: "previous-hash",
}
entry := newFolderEntry(job, folderID, path, updateInfo.UpdatedAt, updateInfo.Hash)
entry := newFolderEntry(job, folderID, path, updateInfo)
Expect(entry.id).To(Equal(folderID))
Expect(entry.job).To(Equal(job))
@ -76,7 +76,7 @@ var _ = Describe("folder_entry", func() {
BeforeEach(func() {
folderID := model.FolderID(lib, path)
entry = newFolderEntry(job, folderID, path, time.Time{}, "")
entry = newFolderEntry(job, folderID, path, model.FolderUpdateInfo{})
})
Describe("hasNoFiles", func() {
@ -457,7 +457,7 @@ var _ = Describe("folder_entry", func() {
// Create new folder entry
folderPath := "music/rock/album"
folderID := model.FolderID(lib, folderPath)
entry := newFolderEntry(job, folderID, folderPath, time.Time{}, "")
entry := newFolderEntry(job, folderID, folderPath, model.FolderUpdateInfo{})
// Initially new and has no files
Expect(entry.isNew()).To(BeTrue())

112
scanner/image_changes.go Normal file
View File

@ -0,0 +1,112 @@
package scanner
import (
"context"
"sync"
"github.com/navidrome/navidrome/consts"
"github.com/navidrome/navidrome/log"
"github.com/navidrome/navidrome/model"
)
// imageChangedFolder records a folder whose image files changed during the scan, so the
// affected albums/artists can be re-enqueued for artwork resolution at the end of phase 1.
type imageChangedFolder struct {
id string
path string
artistImage bool
}
// imageChangeCollector gathers those folders per library while phase 1 persists them, then turns
// them into artwork queue items once.
type imageChangeCollector struct {
libs map[int]model.Library
folders map[int][]imageChangedFolder
ds model.DataStore
mutex sync.Mutex
}
func (c *imageChangeCollector) record(lib model.Library, folder imageChangedFolder) {
c.mutex.Lock()
defer c.mutex.Unlock()
if c.folders == nil {
c.folders = map[int][]imageChangedFolder{}
c.libs = map[int]model.Library{}
}
c.libs[lib.ID] = lib
c.folders[lib.ID] = append(c.folders[lib.ID], folder)
}
// enqueue is best-effort: failures are logged and never fail the scan.
func (c *imageChangeCollector) enqueue(ctx context.Context) {
c.mutex.Lock()
foldersMap, libsMap := c.folders, c.libs
c.folders, c.libs = nil, nil
c.mutex.Unlock()
for libID, folders := range foldersMap {
lib := libsMap[libID]
items, err := c.queueItems(ctx, lib, folders)
if err != nil {
log.Warn(ctx, "Scanner: could not map image changes to artwork items", "lib", lib.Name, err)
continue
}
if len(items) == 0 {
continue
}
if err := c.ds.ArtworkQueue(ctx).Enqueue(items...); err != nil {
log.Warn(ctx, "Scanner: could not enqueue artwork for image changes", "lib", lib.Name, err)
continue
}
log.Debug(ctx, "Scanner: Enqueued artwork resolution for image changes", "lib", lib.Name,
"changedFolders", len(folders), "items", len(items))
}
}
func (c *imageChangeCollector) queueItems(ctx context.Context, lib model.Library,
folders []imageChangedFolder,
) ([]model.ArtworkQueueItem, error) {
folderIDs := make([]string, len(folders))
var artistFolderPaths []string
for i, f := range folders {
folderIDs[i] = f.id
if f.artistImage {
artistFolderPaths = append(artistFolderPaths, f.path)
}
}
var items []model.ArtworkQueueItem
albumIDs, err := c.ds.MediaFile(ctx).GetAlbumIDsByFolder(lib, folderIDs...)
if err != nil {
return nil, err
}
for _, id := range albumIDs {
items = append(items, scanArtworkItem(model.KindAlbumArtwork, id))
}
if len(artistFolderPaths) == 0 {
return items, nil
}
// The resolver climbs to the library root, so the subtree below the folder is the affected set.
// A failure here must not discard the album items already collected.
artistIDs, err := c.ds.Album(ctx).GetSoleAlbumArtistIDsInSubtrees(lib, artistFolderPaths...)
if err != nil {
log.Warn(ctx, "Scanner: could not map image changes to artists", "lib", lib.Name, err)
return items, nil
}
for _, id := range artistIDs {
if id == "" || id == consts.UnknownArtistID || id == consts.VariousArtistsID {
continue
}
items = append(items, scanArtworkItem(model.KindArtistArtwork, id))
}
return items, nil
}
func scanArtworkItem(kind model.Kind, id string) model.ArtworkQueueItem {
return model.ArtworkQueueItem{
ItemKind: kind.Prefix(), ItemID: id, ImageType: model.ImageTypePrimary,
Priority: model.ArtworkPriorityScan,
}
}

View File

@ -45,7 +45,7 @@ func createPhaseFolders(ctx context.Context, state *scanState, ds model.DataStor
jobs = append(jobs, job)
}
return &phaseFolders{jobs: jobs, ctx: ctx, ds: ds, state: state}
return &phaseFolders{jobs: jobs, ctx: ctx, ds: ds, state: state, imageChanges: &imageChangeCollector{ds: ds}}
}
type scanJob struct {
@ -105,7 +105,7 @@ func (j *scanJob) popLastUpdate(folderID string) model.FolderUpdateInfo {
func (j *scanJob) createFolderEntry(path string) *folderEntry {
id := model.FolderID(j.lib, path)
info := j.popLastUpdate(id)
return newFolderEntry(j, id, path, info.UpdatedAt, info.Hash)
return newFolderEntry(j, id, path, info)
}
// phaseFolders represents the first phase of the scanning process, which is responsible
@ -125,6 +125,7 @@ type phaseFolders struct {
ctx context.Context
state *scanState
prevAlbumPIDConf string
imageChanges *imageChangeCollector
}
func (p *phaseFolders) description() string {
@ -169,8 +170,10 @@ func (p *phaseFolders) producer() ppl.Producer[*folderEntry] {
// Check if folder is outdated
if folder.isOutdated() {
if !p.state.fullScan {
if folder.hasNoFiles() && folder.isNew() {
log.Trace(p.ctx, "Scanner: Skipping new folder with no files", "folder", folder.path, "lib", job.lib.Name)
// Ancestor folders need a row even with no files of their own: artwork
// resolution climbs them, and an image added later needs a state to diff.
if folder.isEmpty() && folder.isNew() {
log.Trace(p.ctx, "Scanner: Skipping new empty folder", "folder", folder.path, "lib", job.lib.Name)
continue
}
log.Debug(p.ctx, "Scanner: Detected changes in folder", "folder", folder.path, "lastUpdate", folder.modTime, "lib", job.lib.Name)
@ -197,7 +200,8 @@ func (p *phaseFolders) measure(entry *folderEntry) func() time.Duration {
func (p *phaseFolders) stages() []ppl.Stage[*folderEntry] {
return []ppl.Stage[*folderEntry]{
ppl.NewStage(p.processFolder, ppl.Name("process folder"), ppl.Concurrency(conf.Server.DevScannerThreads)),
ppl.NewStage(p.persistChanges, ppl.Name("persist changes")),
// persistChanges is not reentrant, so it always has to run with concurrency=1
ppl.NewStage(p.persistChanges, ppl.Name("persist changes"), ppl.Concurrency(1)),
ppl.NewStage(p.logFolder, ppl.Name("log results")),
}
}
@ -339,6 +343,15 @@ func (p *phaseFolders) persistChanges(entry *folderEntry) (*folderEntry, error)
albumRepo := tx.Album(p.ctx)
mfRepo := tx.MediaFile(p.ctx)
// A new folder's albums/artists are enqueued below; only pre-existing folders need the diff.
if !entry.isNew() {
if changed, artistImage := entry.imagesChanged(); changed {
p.imageChanges.record(entry.job.lib, imageChangedFolder{
id: entry.id, path: entry.path, artistImage: artistImage,
})
}
}
// Save folder to DB
folder := entry.toFolder()
err := folderRepo.Put(folder)
@ -368,10 +381,7 @@ func (p *phaseFolders) persistChanges(entry *folderEntry) (*folderEntry, error)
return err
}
if entry.artists[i].Name != consts.UnknownArtist && entry.artists[i].Name != consts.VariousArtists {
queueItems = append(queueItems, model.ArtworkQueueItem{
ItemKind: model.KindArtistArtwork.Prefix(), ItemID: entry.artists[i].ID, ImageType: model.ImageTypePrimary,
Priority: model.ArtworkPriorityScan,
})
queueItems = append(queueItems, scanArtworkItem(model.KindArtistArtwork, entry.artists[i].ID))
}
}
@ -383,10 +393,7 @@ func (p *phaseFolders) persistChanges(entry *folderEntry) (*folderEntry, error)
return err
}
if entry.albums[i].Name != consts.UnknownAlbum {
queueItems = append(queueItems, model.ArtworkQueueItem{
ItemKind: model.KindAlbumArtwork.Prefix(), ItemID: entry.albums[i].ID, ImageType: model.ImageTypePrimary,
Priority: model.ArtworkPriorityScan,
})
queueItems = append(queueItems, scanArtworkItem(model.KindAlbumArtwork, entry.albums[i].ID))
}
}
@ -518,6 +525,7 @@ func (p *phaseFolders) finalize(err error) error {
}
return nil
}, "scanner: finalize phaseFolders")
p.imageChanges.enqueue(p.ctx)
return errors.Join(err, errF)
}

View File

@ -278,6 +278,131 @@ var _ = Describe("Scanner", Ordered, func() {
})
})
Context("Library with image files", func() {
var fsys storagetest.FakeFS
image := func(data string) *fstest.MapFile { return &fstest.MapFile{Data: []byte(data)} }
albumID := func(name string) string {
GinkgoHelper()
albums, err := ds.Album(ctx).GetAll(model.QueryOptions{Filters: squirrel.Eq{"album.name": name}})
Expect(err).ToNot(HaveOccurred())
Expect(albums).To(HaveLen(1))
return albums[0].ID
}
artistID := func(name string) string {
GinkgoHelper()
artists, err := ds.Artist(ctx).GetAll(model.QueryOptions{Filters: squirrel.Eq{"artist.name": name}})
Expect(err).ToNot(HaveOccurred())
Expect(artists).To(HaveLen(1))
return artists[0].ID
}
queuedItems := func() []model.ArtworkQueueItem {
GinkgoHelper()
queued, err := ds.ArtworkQueue(ctx).DequeueBatch(1000)
Expect(err).ToNot(HaveOccurred())
return queued
}
queueItemFor := func(kind, id string) OmegaMatcher {
return ContainElement(SatisfyAll(
HaveField("ItemKind", kind),
HaveField("ItemID", id),
HaveField("Priority", model.ArtworkPriorityScan),
))
}
BeforeEach(func() {
revolver := template(_t{"albumartist": "The Beatles", "album": "Revolver", "year": 1966})
wall := template(_t{"albumartist": "Pink Floyd", "album": "The Wall", "year": 1979})
fsys = createFS(fstest.MapFS{
"The Beatles/artist.jpg": image("beatles-artist-v1"),
"The Beatles/Revolver/cover.jpg": image("revolver-cover-v1"),
"The Beatles/Revolver/01 - Taxman.mp3": revolver(track(1, "Taxman")),
"Pink Floyd/The Wall/cover.jpg": image("wall-cover-v1"),
"Pink Floyd/The Wall/CD1/01 - In the Flesh.mp3": wall(track(1, "In the Flesh?")),
"Pink Floyd/The Wall/CD2/01 - Hey You.mp3": wall(track(1, "Hey You")),
})
Expect(runScanner(ctx, true)).To(Succeed())
resolveQueuedArtwork()
})
It("re-enqueues only the album whose cover was replaced in place", func() {
fsys.Add("The Beatles/Revolver/cover.jpg", image("revolver-cover-v2"))
Expect(runScanner(ctx, false)).To(Succeed())
queued := queuedItems()
Expect(queued).To(queueItemFor("al", albumID("Revolver")))
Expect(queued).ToNot(ContainElement(HaveField("ItemID", albumID("The Wall"))))
Expect(queued).ToNot(ContainElement(HaveField("ItemKind", "ar")))
})
It("re-enqueues the album when the cover above its disc folders changes", func() {
fsys.Add("Pink Floyd/The Wall/cover.jpg", image("wall-cover-v2"))
Expect(runScanner(ctx, false)).To(Succeed())
Expect(queuedItems()).To(queueItemFor("al", albumID("The Wall")))
})
It("re-enqueues the album when its cover is removed", func() {
fsys.Remove("The Beatles/Revolver/cover.jpg")
Expect(runScanner(ctx, false)).To(Succeed())
Expect(queuedItems()).To(queueItemFor("al", albumID("Revolver")))
})
It("enqueues the artist when an artist image is added to their folder", func() {
fsys.Add("Pink Floyd/artist.jpg", image("floyd-artist-v1"))
Expect(runScanner(ctx, false)).To(Succeed())
queued := queuedItems()
Expect(queued).To(queueItemFor("ar", artistID("Pink Floyd")))
Expect(queued).ToNot(ContainElement(HaveField("ItemID", artistID("The Beatles"))))
})
It("re-enqueues the artist when their artist image is replaced in place", func() {
fsys.Add("The Beatles/artist.jpg", image("beatles-artist-v2"))
Expect(runScanner(ctx, false)).To(Succeed())
Expect(queuedItems()).To(queueItemFor("ar", artistID("The Beatles")))
})
It("enqueues every artist under the folder when a shared artist image is added", func() {
fsys.Add("artist.png", image("shared-artist-v1"))
Expect(runScanner(ctx, false)).To(Succeed())
queued := queuedItems()
Expect(queued).To(queueItemFor("ar", artistID("The Beatles")))
Expect(queued).To(queueItemFor("ar", artistID("Pink Floyd")))
})
It("enqueues the artist when an image lands in a folder first seen by a quick scan", func() {
// A quick scan must persist an artist folder that holds only subfolders, or the
// artist.jpg added later has no previous state to diff against.
kraftwerk := template(_t{"albumartist": "Kraftwerk", "album": "Autobahn", "year": 1974})
files := fsys.MapFS
files["Kraftwerk/Autobahn/01 - Autobahn.mp3"] = kraftwerk(track(1, "Autobahn"))
fsys.SetFiles(files)
Expect(runScanner(ctx, false)).To(Succeed())
resolveQueuedArtwork()
fsys.Add("Kraftwerk/artist.jpg", image("kraftwerk-artist-v1"))
Expect(runScanner(ctx, false)).To(Succeed())
Expect(queuedItems()).To(queueItemFor("ar", artistID("Kraftwerk")))
})
It("does not enqueue anything on a repeat full scan with no image changes", func() {
Expect(runScanner(ctx, true)).To(Succeed())
Expect(queuedItems()).To(BeEmpty())
})
})
Context("Artist with atomic non-ASCII letters, 'GØGGS'", func() {
BeforeEach(func() {
goggs := template(_t{"albumartist": "GØGGS", "album": "Pre Strike Sweep", "year": 2018})