navidrome/scanner/watcher_test.go
Deluan Quintão 7736bbb545
fix(cache): write the completion marker before closing the cache writer (#5927)
* fix(cache): write the completion marker before closing the cache writer

Readers of an in-progress cache write see EOF the moment the writer closes,
but the .complete marker was created after the close, on the background
goroutine — so a fully-read stream did not mean the cache was done touching
disk. The new artwork precache spec ends right at EOF, and its
GinkgoT().TempDir() cleanup raced the marker creation, failing the Windows CI
job with 'unlinkat ...: The directory is not empty' (the race also reproduces
on macOS, 2 of 3 runs, with the tightened test).

Writing the marker after a clean copy but before Close makes reader-EOF imply
every on-disk write for the entry is finished. A failed writer Close still
invalidates the entry, which removes both the marker and the data file. The
existing marker test now asserts the marker exists immediately at EOF instead
of Eventually.

* fix(artwork): never dispatch queue items after the drain context is cancelled

The 10x Windows stress run for the previous commit surfaced a second flake in
the same package: 'leaves undispatched items queued when cancelled mid-batch'
lost row alc7 in 4 of 10 runs. In drain, when a semaphore slot is free and the
context is already cancelled, both cases of the blocking select are ready and
Go picks one at random — so a cancelled drain could still dispatch items. A
non-blocking Done check before the select gives cancellation priority.

The race was invisible on Linux/macOS only by accident: the spec seeded the
album repo with a single album (each SetData overwrote the last), so only the
final row (alc7) resolved to absent and got deleted when dispatched; the
others fell on the retry path and survived. Nanosecond enqueue timestamps
made alc0 always first out of the mock dequeue, masking the race, while
Windows' coarse clock ties the timestamps and randomizes the order. The spec
now seeds all eight albums, which made the race reproduce locally on the
first try (row alc0) and now guards the fix on every platform.

* test: give cache-init waits a 10s timeout for loaded CI runners

A 10x parallel Windows stress run timed out one artwork spec in BeforeEach:
the FileCache init goroutine (mkdir + reload walk) took over Gomega's default
1s Eventually timeout under shared-runner disk contention. Bump the three
identical init waits (two artwork suites and the utils/cache helper) to 10s.

* test(scanner): widen watcher debounce margins for loaded CI runners

The watcher debouncing spec asserts 'no scan yet' inside 20ms Consistently
windows while the debounce wait was only 50ms — a 2.5x margin that a loaded
Windows runner blows through by delaying the timer-reset notification, firing
the scan early (failed all three FlakeAttempts in a 10x stress run). Raise the
test debounce wait to 200ms (10x the observation windows) and the scan-fired
Eventually timeouts to 2s to match.

* refactor(artwork): collapse drain cancellation into a single exit path

Replace the non-blocking ctx pre-check plus duplicated select exit with one
select and a ctx.Err() check after it. Besides removing the duplication, this
closes the residual race: a cancellation landing between the two selects could
still let the blocking select randomly pick the free semaphore slot and
dispatch the item. Now a dispatch is only possible when the context was live
after slot acquisition.

* style: trim flaky-test fix comments to single lines

Compress each two-line comment added by this PR to the one line that carries
the invariant; drop the narration around it.
2026-08-10 14:10:05 -04:00

544 lines
17 KiB
Go

package scanner
import (
"context"
"io/fs"
"path/filepath"
"testing/fstest"
"time"
"github.com/navidrome/navidrome/conf"
"github.com/navidrome/navidrome/conf/configtest"
"github.com/navidrome/navidrome/core/storage/storagetest"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/tests"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
var _ = Describe("Watcher", func() {
var ctx context.Context
var cancel context.CancelFunc
var mockScanner *tests.MockScanner
var mockDS *tests.MockDataStore
var w *watcher
var lib *model.Library
BeforeEach(func() {
DeferCleanup(configtest.SetupConfig())
// Must dwarf the 20ms Consistently windows below, or a loaded runner's delayed timer reset flakes the debouncing spec.
conf.Server.Scanner.WatcherWait = 200 * time.Millisecond
ctx, cancel = context.WithCancel(GinkgoT().Context())
DeferCleanup(cancel)
// Use a fake storage scheme: watchLibrary goroutines spawned by Run/Watch are not
// joined on spec teardown, and the real file:// storage reads conf.Server on
// construction, racing with the configtest cleanup that restores the config.
storagetest.Register("fake-watcher", &storagetest.FakeFS{})
lib = &model.Library{
ID: 1,
Name: "Test Library",
Path: "fake-watcher:///test/library",
}
// Set up mocks
mockScanner = tests.NewMockScanner()
mockDS = &tests.MockDataStore{}
mockLibRepo := &tests.MockLibraryRepo{}
mockLibRepo.SetData(model.Libraries{*lib})
mockDS.MockedLibrary = mockLibRepo
// Create a new watcher instance (not singleton) for testing
w = &watcher{
ds: mockDS,
scanner: mockScanner,
triggerWait: conf.Server.Scanner.WatcherWait,
watcherNotify: make(chan scanNotification, 10),
libraryWatchers: make(map[int]*libraryWatcherInstance),
mainCtx: ctx,
}
})
Describe("Watch before Run", func() {
It("returns nil and does not panic when mainCtx is nil", func() {
w.mainCtx = nil
err := w.Watch(ctx, lib)
Expect(err).ToNot(HaveOccurred())
Expect(w.libraryWatchers).To(BeEmpty())
})
})
Describe("Target Collection and Deduplication", func() {
BeforeEach(func() {
// Start watcher in background
go func() {
_ = w.Run(ctx)
}()
// Give watcher time to initialize
time.Sleep(10 * time.Millisecond)
})
It("creates separate targets for different folders", FlakeAttempts(3), func() {
// Send notifications for different folders
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1"}
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist2"}
// Wait for a scan that collected both targets
Eventually(func() []model.ScanTarget {
calls := mockScanner.GetScanFoldersCalls()
if len(calls) == 0 {
return nil
}
return calls[0].Targets
}, 2*time.Second, 10*time.Millisecond).Should(HaveLen(2))
// Verify targets
calls := mockScanner.GetScanFoldersCalls()
folderPaths := make(map[string]bool)
for _, target := range calls[0].Targets {
Expect(target.LibraryID).To(Equal(1))
folderPaths[target.FolderPath] = true
}
Expect(folderPaths).To(HaveKey("artist1"))
Expect(folderPaths).To(HaveKey("artist2"))
})
It("handles different folder paths correctly", func() {
// Send notification for nested folder
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1/album1"}
// Wait for watcher to process and trigger scan
Eventually(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 2*time.Second, 10*time.Millisecond).Should(Equal(1))
// Verify the target
calls := mockScanner.GetScanFoldersCalls()
Expect(calls).To(HaveLen(1))
Expect(calls[0].Targets).To(HaveLen(1))
Expect(calls[0].Targets[0].FolderPath).To(Equal("artist1/album1"))
})
It("deduplicates folder and file within same folder", func() {
// Send multiple notifications for the same folder
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1/album1"}
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1/album1"}
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1/album1"}
// Wait for watcher to process and trigger scan
Eventually(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 2*time.Second, 10*time.Millisecond).Should(Equal(1))
// Verify only one target despite multiple file/folder changes
calls := mockScanner.GetScanFoldersCalls()
Expect(calls).To(HaveLen(1))
Expect(calls[0].Targets).To(HaveLen(1))
Expect(calls[0].Targets[0].FolderPath).To(Equal("artist1/album1"))
})
})
Describe("Timer Behavior", func() {
BeforeEach(func() {
// Start watcher in background
go func() {
_ = w.Run(ctx)
}()
// Give watcher time to initialize
time.Sleep(10 * time.Millisecond)
})
It("resets timer on each change (debouncing)", FlakeAttempts(3), func() {
// Send first notification
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1"}
// Verify no scan fires during a window shorter than the debounce wait
Consistently(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 20*time.Millisecond, 5*time.Millisecond).Should(Equal(0))
// Send another notification (resets timer)
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1"}
// Again, no scan should fire within a short window
Consistently(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 20*time.Millisecond, 5*time.Millisecond).Should(Equal(0))
// Now wait for the debounce timer to expire and trigger scan
Eventually(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 2*time.Second, 10*time.Millisecond).Should(Equal(1))
})
It("triggers scan after quiet period", func() {
// Send notification
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1"}
// No scan immediately
Expect(mockScanner.GetScanFoldersCallCount()).To(Equal(0))
// Wait for quiet period
Eventually(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 2*time.Second, 10*time.Millisecond).Should(Equal(1))
})
})
Describe("Empty and Root Paths", func() {
BeforeEach(func() {
// Start watcher in background
go func() {
_ = w.Run(ctx)
}()
// Give watcher time to initialize
time.Sleep(10 * time.Millisecond)
})
It("handles empty folder path (library root)", func() {
// Send notification with empty folder path
w.watcherNotify <- scanNotification{Library: lib, FolderPath: ""}
// Wait for scan
Eventually(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 2*time.Second, 10*time.Millisecond).Should(Equal(1))
// Should scan the library root
calls := mockScanner.GetScanFoldersCalls()
Expect(calls).To(HaveLen(1))
Expect(calls[0].Targets).To(HaveLen(1))
Expect(calls[0].Targets[0].FolderPath).To(Equal(""))
})
It("deduplicates empty and dot paths", func() {
// Send notifications with empty and dot paths
w.watcherNotify <- scanNotification{Library: lib, FolderPath: ""}
w.watcherNotify <- scanNotification{Library: lib, FolderPath: ""}
// Wait for scan
Eventually(func() int {
return mockScanner.GetScanFoldersCallCount()
}, 2*time.Second, 10*time.Millisecond).Should(Equal(1))
// Should have only one target
calls := mockScanner.GetScanFoldersCalls()
Expect(calls).To(HaveLen(1))
Expect(calls[0].Targets).To(HaveLen(1))
})
})
Describe("Multiple Libraries", func() {
var lib2 *model.Library
BeforeEach(func() {
// Create second library
lib2 = &model.Library{
ID: 2,
Name: "Test Library 2",
Path: "fake-watcher:///test/library2",
}
mockLibRepo := mockDS.MockedLibrary.(*tests.MockLibraryRepo)
mockLibRepo.SetData(model.Libraries{*lib, *lib2})
// Start watcher in background
go func() {
_ = w.Run(ctx)
}()
// Give watcher time to initialize
time.Sleep(10 * time.Millisecond)
})
It("creates separate targets for different libraries", func() {
// Send notifications for both libraries
w.watcherNotify <- scanNotification{Library: lib, FolderPath: "artist1"}
w.watcherNotify <- scanNotification{Library: lib2, FolderPath: "artist2"}
// Wait for a scan that collected both targets
Eventually(func() []model.ScanTarget {
calls := mockScanner.GetScanFoldersCalls()
if len(calls) == 0 {
return nil
}
return calls[0].Targets
}, 2*time.Second, 10*time.Millisecond).Should(HaveLen(2))
// Verify library IDs are different
calls := mockScanner.GetScanFoldersCalls()
libraryIDs := make(map[int]bool)
for _, target := range calls[0].Targets {
libraryIDs[target.LibraryID] = true
}
Expect(libraryIDs).To(HaveKey(1))
Expect(libraryIDs).To(HaveKey(2))
})
})
Describe(".ndignore handling", func() {
var ctx context.Context
var cancel context.CancelFunc
var w *watcher
var mockFS *mockMusicFS
var lib *model.Library
var eventChan chan string
var absLibPath string
BeforeEach(func() {
ctx, cancel = context.WithCancel(GinkgoT().Context())
DeferCleanup(cancel)
// Set up library
var err error
absLibPath, err = filepath.Abs(".")
Expect(err).NotTo(HaveOccurred())
lib = &model.Library{
ID: 1,
Name: "Test Library",
Path: absLibPath,
}
// Create watcher with notification channel
w = &watcher{
watcherNotify: make(chan scanNotification, 10),
}
eventChan = make(chan string, 10)
})
// Helper to send an event - converts relative path to absolute
sendEvent := func(relativePath string) {
path := filepath.Join(absLibPath, relativePath)
eventChan <- path
}
// Helper to start the real event processing loop
startEventProcessing := func() {
go func() {
defer GinkgoRecover()
// Call the actual processLibraryEvents method - testing the real implementation!
_ = w.processLibraryEvents(ctx, lib, mockFS, eventChan, absLibPath)
}()
}
Context("when a folder matching .ndignore is deleted", func() {
BeforeEach(func() {
// Create filesystem with .ndignore containing _TEMP pattern
// The deleted folder (_TEMP) will NOT exist in the filesystem
mockFS = &mockMusicFS{
FS: fstest.MapFS{
"rock": &fstest.MapFile{Mode: fs.ModeDir},
"rock/.ndignore": &fstest.MapFile{Data: []byte("_TEMP\n")},
"rock/valid_album": &fstest.MapFile{Mode: fs.ModeDir},
"rock/valid_album/track.mp3": &fstest.MapFile{Data: []byte("audio")},
},
}
})
It("should NOT send scan notification when deleted folder matches .ndignore", func() {
startEventProcessing()
// Simulate deletion event for rock/_TEMP
sendEvent("rock/_TEMP")
// Wait a bit to ensure event is processed
time.Sleep(50 * time.Millisecond)
// No notification should have been sent
Consistently(eventChan, 100*time.Millisecond).Should(BeEmpty())
})
It("should send scan notification for valid folder deletion", func() {
startEventProcessing()
// Simulate deletion event for rock/other_folder (not in .ndignore and doesn't exist)
// Since it doesn't exist in mockFS, resolveFolderPath will walk up to "rock"
sendEvent("rock/other_folder")
// Should receive notification for parent folder
Eventually(w.watcherNotify, 200*time.Millisecond).Should(Receive(Equal(scanNotification{
Library: lib,
FolderPath: "rock",
})))
})
})
Context("with nested folder patterns", func() {
BeforeEach(func() {
mockFS = &mockMusicFS{
FS: fstest.MapFS{
"music": &fstest.MapFile{Mode: fs.ModeDir},
"music/.ndignore": &fstest.MapFile{Data: []byte("**/temp\n**/cache\n")},
"music/rock": &fstest.MapFile{Mode: fs.ModeDir},
"music/rock/artist": &fstest.MapFile{Mode: fs.ModeDir},
},
}
})
It("should NOT send notification when nested ignored folder is deleted", func() {
tests.SkipOnWindows("path separator bug (#TBD-path-sep-scanner)")
startEventProcessing()
// Simulate deletion of music/rock/artist/temp (matches **/temp)
sendEvent("music/rock/artist/temp")
// Wait to ensure event is processed
time.Sleep(50 * time.Millisecond)
// No notification should be sent
Expect(w.watcherNotify).To(BeEmpty(), "Expected no scan notification for nested ignored folder")
})
It("should send notification for non-ignored nested folder", func() {
tests.SkipOnWindows("path separator bug (#TBD-path-sep-scanner)")
startEventProcessing()
// Simulate change in music/rock/artist (doesn't match any pattern)
sendEvent("music/rock/artist")
// Should receive notification
Eventually(w.watcherNotify, 200*time.Millisecond).Should(Receive(Equal(scanNotification{
Library: lib,
FolderPath: "music/rock/artist",
})))
})
})
Context("with file events in ignored folders", func() {
BeforeEach(func() {
mockFS = &mockMusicFS{
FS: fstest.MapFS{
"rock": &fstest.MapFile{Mode: fs.ModeDir},
"rock/.ndignore": &fstest.MapFile{Data: []byte("_TEMP\n")},
},
}
})
It("should NOT send notification for file changes in ignored folders", func() {
tests.SkipOnWindows("path separator bug (#TBD-path-sep-scanner)")
startEventProcessing()
// Simulate file change in rock/_TEMP/file.mp3
sendEvent("rock/_TEMP/file.mp3")
// Wait to ensure event is processed
time.Sleep(50 * time.Millisecond)
// No notification should be sent
Expect(w.watcherNotify).To(BeEmpty(), "Expected no scan notification for file in ignored folder")
})
})
})
})
var _ = Describe("isIgnoredPath", func() {
BeforeEach(func() {
DeferCleanup(configtest.SetupConfig())
})
Context("with IgnoreDotFolders enabled (default)", func() {
BeforeEach(func() {
conf.Server.Scanner.IgnoreDotFolders = true
})
DescribeTable("returns expected result",
func(p string, expected bool) {
Expect(isIgnoredPath(context.Background(), nil, filepath.FromSlash(p))).To(Equal(expected))
},
Entry("media file in normal folder", "rock/Album/track.mp3", false),
Entry("dot-prefixed media file", "rock/Album/.hidden.mp3", true),
Entry("media file inside a dot-folder", "rock/.Hidden Album/track.mp3", true),
Entry("media file inside a blocklisted folder", "rock/.streams/stream.mp3", true),
Entry("media file inside .git", "rock/.git/track.mp3", true),
Entry("dot-folder itself", "rock/.Hidden Album", true),
Entry("normal folder itself", "rock/Album", false),
Entry(".DS_Store file", "rock/Album/.DS_Store", true),
)
})
Context("with IgnoreDotFolders disabled", func() {
BeforeEach(func() {
conf.Server.Scanner.IgnoreDotFolders = false
})
DescribeTable("returns expected result",
func(p string, expected bool) {
Expect(isIgnoredPath(context.Background(), nil, filepath.FromSlash(p))).To(Equal(expected))
},
Entry("media file inside a dot-folder is allowed", "rock/.Hidden Album/track.mp3", false),
Entry("dot-prefixed media file is still ignored", "rock/Album/.hidden.mp3", true),
Entry("dot-folder itself is allowed", "rock/.Hidden Album", false),
Entry("blocklisted folder still ignored", "rock/.streams/stream.mp3", true),
Entry(".git still ignored", "rock/.git/config", true),
)
})
})
var _ = Describe("resolveFolderPath", func() {
var mockFS fs.FS
BeforeEach(func() {
// Create a mock filesystem with some directories and files
mockFS = fstest.MapFS{
"artist1": &fstest.MapFile{Mode: fs.ModeDir},
"artist1/album1": &fstest.MapFile{Mode: fs.ModeDir},
"artist1/album1/track1.mp3": &fstest.MapFile{Data: []byte("audio")},
"artist1/album1/track2.mp3": &fstest.MapFile{Data: []byte("audio")},
"artist1/album2": &fstest.MapFile{Mode: fs.ModeDir},
"artist1/album2/song.flac": &fstest.MapFile{Data: []byte("audio")},
"artist2": &fstest.MapFile{Mode: fs.ModeDir},
"artist2/cover.jpg": &fstest.MapFile{Data: []byte("image")},
}
})
It("returns directory path when given a directory", func() {
result := resolveFolderPath(mockFS, "artist1/album1")
Expect(result).To(Equal("artist1/album1"))
})
It("walks up to parent directory when given a file path", func() {
tests.SkipOnWindows("path separator bug (#TBD-path-sep-scanner)")
result := resolveFolderPath(mockFS, "artist1/album1/track1.mp3")
Expect(result).To(Equal("artist1/album1"))
})
It("walks up multiple levels if needed", func() {
tests.SkipOnWindows("path separator bug (#TBD-path-sep-scanner)")
result := resolveFolderPath(mockFS, "artist1/album1/nonexistent/file.mp3")
Expect(result).To(Equal("artist1/album1"))
})
It("returns empty string for non-existent paths at root", func() {
result := resolveFolderPath(mockFS, "nonexistent/path/file.mp3")
Expect(result).To(Equal(""))
})
It("returns empty string for dot path", func() {
result := resolveFolderPath(mockFS, ".")
Expect(result).To(Equal(""))
})
It("returns empty string for empty path", func() {
result := resolveFolderPath(mockFS, "")
Expect(result).To(Equal(""))
})
It("handles nested file paths correctly", func() {
tests.SkipOnWindows("path separator bug (#TBD-path-sep-scanner)")
result := resolveFolderPath(mockFS, "artist1/album2/song.flac")
Expect(result).To(Equal("artist1/album2"))
})
It("resolves to top-level directory", func() {
result := resolveFolderPath(mockFS, "artist2/cover.jpg")
Expect(result).To(Equal("artist2"))
})
})