navidrome/persistence/library_repository.go
Deluan Quintão cc315dcc8c
perf(db): keep query planner statistics trustworthy with full ANALYZE (#5740)
* perf(db): keep query planner statistics trustworthy with full ANALYZE

PRAGMA optimize's internal ANALYZE runs with a limited analysis budget
(~2000 rows) that writes wrong sqlite_stat1 entries for low-cardinality
indexes: on a 96K-track library it claimed (missing, library_id) narrows
to ~2000 rows when it matches the whole table. The planner then prefers
that index over the sort index and falls back to a full-table temp
B-tree sort per request, turning paginated song listings into
multi-second queries (reproduced at 5.5s on real hardware; ~90x slower
than with correct stats). Every index-creating migration re-triggered
the poisoning via the post-migration optimize, and the daily optimizer
could re-trigger it on large library changes. Setting analysis_limit on
the connection does not help: optimize ignores it.

Run a plain full ANALYZE instead: after migrations with schema changes,
and in db.Optimize (daily schedule and scan-end). Stats are stored in
the database file, so one connection suffices and the per-connection
pool loop is gone. The Optimize call at shutdown is removed: stats are
maintained at migration/scan/daily points, and an ANALYZE during
shutdown only delays it and races container stop timeouts.

* perf(db): drop startup PRAGMA optimize that re-poisons planner stats

The startup PRAGMA optimize=0x10002 runs SQLite's budget-limited internal
ANALYZE (bit 0x02), which writes truncated sqlite_stat1 rows for
low-cardinality indexes -- the exact statistics-poisoning this PR set out to
eliminate. Because DevOptimizeDB defaults to true, a restart with no pending
migrations would re-poison the planner until the next scan or daily Optimize.

Remove it: statistics are already refreshed with a full ANALYZE after
schema-changing migrations (Init) and via Optimize at scan-end and on the daily
schedule, so nothing on the startup path needs to touch them. Also clarify that
Optimize is a no-op unless DevOptimizeDB is enabled.

* chore(db): remove the DevOptimizeDB flag and skip Optimize on quick scans

The flag only gated the optimize/ANALYZE maintenance calls and there is
no reason to leave planner statistics unmaintained; the guards are gone
along with the flag. The scan-end Optimize now runs only after full
scans — quick scans barely move the statistics, and the daily schedule
covers drift.

* style(scanner): drop redundant comment in runOptimize

* chore(persistence): drop the no-op PRAGMA optimize from ScanEnd

Mask 0x10000 only selects candidate tables by size change; without the
0x02 action bit optimize does nothing (verified: sqlite_stat1 stays
stale after a 100x table growth). The scan-end statistics refresh is
db.Optimize's full ANALYZE, and the expression-collation-index concern
the old comment guarded against no longer applies.

* fix(scanner): run the post-scan ANALYZE in the server process

With the external scanner (the default), the scan pipeline runs in a
subprocess, so its ANALYZE was invisible to the server: SQLite loads
sqlite_stat1 into the process's shared schema cache, and an ANALYZE from
another process does not refresh it — verified with the production DSN
that even brand-new pool connections keep planning with the old
statistics until the server restarts. An in-process ANALYZE, by
contrast, is immediately visible to every pooled connection through the
same shared cache.

Move the full-scan Optimize from the scanner pipeline to the scan
controller, which always runs in the server process.

* fix(scanner): honor promoted full scans in the optimize gate

A quick scan resuming an interrupted full scan is promoted inside the
scanner (possibly in a subprocess); mirror the promotion in the
controller so the post-scan ANALYZE isn't skipped.

* refactor: apply cleanup review findings

- drop forceFullRescan's inline ANALYZE: Init already runs a full
  ANALYZE after any migration batch with schema changes, so upgrades
  including a full-rescan migration analyzed the whole DB twice
- resumingFullScan uses a filtered CountAll instead of fetching and
  scanning all libraries
- document why CallScan (CLI) deliberately skips the post-scan Optimize

* perf(db): make planner analysis maintenance resilient

Check analysis freshness every 30 minutes and refresh statistics when the last successful run is over 24 hours old or a scan marked them pending. Persist successful analysis state, retry skipped or failed maintenance, coordinate checks with scans, and cover standalone CLI full scans.

* perf(db): avoid analyzing routine quick-scan changes

Reserve pending analysis for full scans, unscanned libraries, and retry state. Incremental quick scans now rely on the 24-hour freshness window instead of triggering a full ANALYZE at the next maintenance check.

* fix(scan): analyze resumed full scans in CLI

* fix(db): back off failed analysis retries

* feat(db): allow disabling scheduled analysis

* test(db): remove redundant analysis coverage

* refactor(db): split ANALYZE maintenance into optimize.go and dedupe call sites

- move query-planner statistics code from db.go to its own optimize.go
  (and matching optimize_test.go)
- log ANALYZE elapsed time inside Optimize/OptimizeIfNeeded instead of
  repeating the timing block at every call site
- drop the LastDBAnalyzeAttemptAt write on success: it is only read
  while failures >= 1, and every failure rewrites it first
- extract runPostScanAnalysis (cmd) and anyIncludedLibrary (scanner)
  helpers
2026-07-13 12:04:29 -04:00

348 lines
9.3 KiB
Go

package persistence
import (
"context"
"fmt"
"strconv"
"sync"
"time"
. "github.com/Masterminds/squirrel"
"github.com/deluan/rest"
"github.com/navidrome/navidrome/conf"
"github.com/navidrome/navidrome/log"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/utils/run"
"github.com/pocketbase/dbx"
)
type libraryRepository struct {
sqlRepository
}
var (
libCache = map[int]string{}
libLock sync.RWMutex
)
func NewLibraryRepository(ctx context.Context, db dbx.Builder) model.LibraryRepository {
r := &libraryRepository{}
r.ctx = ctx
r.db = db
r.registerModel(&model.Library{}, nil)
return r
}
func (r *libraryRepository) Get(id int) (*model.Library, error) {
sq := r.newSelect().Columns("*").Where(Eq{"id": id})
var res model.Library
err := r.queryOne(sq, &res)
return &res, err
}
func (r *libraryRepository) GetPath(id int) (string, error) {
l := func() string {
libLock.RLock()
defer libLock.RUnlock()
if l, ok := libCache[id]; ok {
return l
}
return ""
}()
if l != "" {
return l, nil
}
libLock.Lock()
defer libLock.Unlock()
libs, err := r.GetAll()
if err != nil {
log.Error(r.ctx, "Error loading libraries from DB", err)
return "", err
}
for _, l := range libs {
libCache[l.ID] = l.Path
}
if l, ok := libCache[id]; ok {
return l, nil
} else {
return "", model.ErrNotFound
}
}
func (r *libraryRepository) Put(l *model.Library) error {
if l.ID == model.DefaultLibraryID {
currentLib, err := r.Get(1)
// if we are creating it, it's ok.
if err == nil { // it exists, so we are updating it
if currentLib.Path != l.Path {
return fmt.Errorf("%w: path for library with ID 1 cannot be changed", model.ErrValidation)
}
}
}
var err error
l.UpdatedAt = time.Now()
if l.ID == 0 {
// Insert with autoassigned ID
l.CreatedAt = time.Now()
err = r.db.Model(l).Insert()
} else {
// Try to update first
cols := map[string]any{
"name": l.Name,
"path": l.Path,
"remote_path": l.RemotePath,
"default_new_users": l.DefaultNewUsers,
"updated_at": l.UpdatedAt,
}
sq := Update(r.tableName).SetMap(cols).Where(Eq{"id": l.ID})
rowsAffected, updateErr := r.executeSQL(sq)
if updateErr != nil {
return updateErr
}
// If no rows were affected, the record doesn't exist, so insert it
if rowsAffected == 0 {
l.CreatedAt = time.Now()
l.UpdatedAt = time.Now()
err = r.db.Model(l).Insert()
}
}
if err != nil {
return err
}
// Auto-assign all libraries to all admin users
sql := Expr(`
INSERT INTO user_library (user_id, library_id)
SELECT u.id, l.id
FROM user u
CROSS JOIN library l
WHERE u.is_admin = true
ON CONFLICT (user_id, library_id) DO NOTHING;`,
)
if _, err = r.executeSQL(sql); err != nil {
return fmt.Errorf("failed to assign library to admin users: %w", err)
}
libLock.Lock()
defer libLock.Unlock()
libCache[l.ID] = l.Path
return nil
}
// TODO Remove this method when we have a proper UI to add libraries
// This is a temporary method to store the music folder path from the config in the DB
func (r *libraryRepository) StoreMusicFolder() error {
sq := Update(r.tableName).Set("path", conf.Server.MusicFolder).
Set("updated_at", time.Now()).
Where(Eq{"id": model.DefaultLibraryID})
_, err := r.executeSQL(sq)
if err != nil {
libLock.Lock()
defer libLock.Unlock()
libCache[model.DefaultLibraryID] = conf.Server.MusicFolder
}
return err
}
func (r *libraryRepository) AddArtist(id int, artistID string) error {
sq := Insert("library_artist").Columns("library_id", "artist_id").Values(id, artistID).
Suffix(`on conflict(library_id, artist_id) do nothing`)
_, err := r.executeSQL(sq)
if err != nil {
return err
}
return nil
}
func (r *libraryRepository) ScanBegin(id int, fullScan bool) error {
sq := Update(r.tableName).
Set("last_scan_started_at", time.Now()).
Set("full_scan_in_progress", fullScan).
Where(Eq{"id": id})
_, err := r.executeSQL(sq)
return err
}
func (r *libraryRepository) ScanEnd(id int) error {
sq := Update(r.tableName).
Set("last_scan_at", time.Now()).
Set("full_scan_in_progress", false).
Set("last_scan_started_at", time.Time{}).
Where(Eq{"id": id})
_, err := r.executeSQL(sq)
return err
}
func (r *libraryRepository) ScanInProgress() (bool, error) {
query := r.newSelect().Where(NotEq{"last_scan_started_at": time.Time{}})
count, err := r.count(query)
return count > 0, err
}
func (r *libraryRepository) RefreshStats(id int) error {
var songsRes, albumsRes, artistsRes, foldersRes, filesRes, missingRes struct{ Count int64 }
var sizeRes struct{ Sum int64 }
var durationRes struct{ Sum float64 }
err := run.Parallel(
func() error {
return r.queryOne(Select("count(*) as count").From("media_file").Where(Eq{"library_id": id, "missing": false}), &songsRes)
},
func() error {
return r.queryOne(Select("count(*) as count").From("album").Where(Eq{"library_id": id, "missing": false}), &albumsRes)
},
func() error {
return r.queryOne(Select("count(*) as count").From("library_artist la").
Join("artist a on la.artist_id = a.id").
Where(Eq{"la.library_id": id, "a.missing": false}), &artistsRes)
},
func() error {
return r.queryOne(Select("count(*) as count").From("folder").
Where(And{
Eq{"library_id": id, "missing": false},
Gt{"num_audio_files": 0},
}), &foldersRes)
},
func() error {
return r.queryOne(Select("ifnull(sum(num_audio_files + num_playlists + json_array_length(image_files)),0) as count").
From("folder").Where(Eq{"library_id": id, "missing": false}), &filesRes)
},
func() error {
return r.queryOne(Select("count(*) as count").From("media_file").Where(Eq{"library_id": id, "missing": true}), &missingRes)
},
func() error {
return r.queryOne(Select("ifnull(sum(size),0) as sum").From("album").Where(Eq{"library_id": id, "missing": false}), &sizeRes)
},
func() error {
return r.queryOne(Select("ifnull(sum(duration),0) as sum").From("album").Where(Eq{"library_id": id, "missing": false}), &durationRes)
},
)()
if err != nil {
return err
}
sq := Update(r.tableName).
Set("total_songs", songsRes.Count).
Set("total_albums", albumsRes.Count).
Set("total_artists", artistsRes.Count).
Set("total_folders", foldersRes.Count).
Set("total_files", filesRes.Count).
Set("total_missing_files", missingRes.Count).
Set("total_size", sizeRes.Sum).
Set("total_duration", durationRes.Sum).
Set("updated_at", time.Now()).
Where(Eq{"id": id})
_, err = r.executeSQL(sq)
return err
}
func (r *libraryRepository) Delete(id int) error {
if !loggedUser(r.ctx).IsAdmin {
return model.ErrNotAuthorized
}
if id == 1 {
return fmt.Errorf("%w: library with ID 1 cannot be deleted", model.ErrValidation)
}
err := r.delete(Eq{"id": id})
if err != nil {
return err
}
// The cascade above can drop an artist's last library_artist row; reconcile any such orphans.
if err := NewArtistRepository(r.ctx, r.db).(*artistRepository).markOrphansMissing(); err != nil {
return fmt.Errorf("marking orphaned artists missing after deleting library %d: %w", id, err)
}
// Clear cache entry for this library only if DB operation was successful
libLock.Lock()
defer libLock.Unlock()
delete(libCache, id)
// Clean up orphaned plugin references for the deleted library
if err := cleanupPluginLibraryReferences(r.db, id); err != nil {
log.Error(r.ctx, "Failed to cleanup plugin library references", "libraryID", id, err)
}
return nil
}
func (r *libraryRepository) GetAll(ops ...model.QueryOptions) (model.Libraries, error) {
sq := r.newSelect(ops...).Columns("*")
res := model.Libraries{}
err := r.queryAll(sq, &res)
return res, err
}
func (r *libraryRepository) CountAll(ops ...model.QueryOptions) (int64, error) {
sq := r.newSelect(ops...)
return r.count(sq)
}
// User-library association methods
func (r *libraryRepository) GetUsersWithLibraryAccess(libraryID int) (model.Users, error) {
sel := Select("u.*").
From("user u").
Join("user_library ul ON u.id = ul.user_id").
Where(Eq{"ul.library_id": libraryID}).
OrderBy("u.name")
var res model.Users
err := r.queryAll(sel, &res)
return res, err
}
// REST interface methods
func (r *libraryRepository) Count(options ...rest.QueryOptions) (int64, error) {
return r.CountAll(r.parseRestOptions(r.ctx, options...))
}
func (r *libraryRepository) Read(id string) (any, error) {
idInt, err := strconv.Atoi(id)
if err != nil {
log.Trace(r.ctx, "invalid library id: %s", id, err)
return nil, rest.ErrNotFound
}
return r.Get(idInt)
}
func (r *libraryRepository) ReadAll(options ...rest.QueryOptions) (any, error) {
return r.GetAll(r.parseRestOptions(r.ctx, options...))
}
func (r *libraryRepository) EntityName() string {
return "library"
}
func (r *libraryRepository) NewInstance() any {
return &model.Library{}
}
func (r *libraryRepository) Save(entity any) (string, error) {
lib := entity.(*model.Library)
lib.ID = 0 // Reset ID to ensure we create a new library
err := r.Put(lib)
if err != nil {
return "", err
}
return strconv.Itoa(lib.ID), nil
}
func (r *libraryRepository) Update(id string, entity any, cols ...string) error {
lib := entity.(*model.Library)
idInt, err := strconv.Atoi(id)
if err != nil {
return fmt.Errorf("invalid library ID: %s", id)
}
lib.ID = idInt
return r.Put(lib)
}
var _ model.LibraryRepository = (*libraryRepository)(nil)
var _ rest.Repository = (*libraryRepository)(nil)