mirror of
https://github.com/navidrome/navidrome.git
synced 2026-08-01 07:21:17 +00:00
* perf(db): keep query planner statistics trustworthy with full ANALYZE PRAGMA optimize's internal ANALYZE runs with a limited analysis budget (~2000 rows) that writes wrong sqlite_stat1 entries for low-cardinality indexes: on a 96K-track library it claimed (missing, library_id) narrows to ~2000 rows when it matches the whole table. The planner then prefers that index over the sort index and falls back to a full-table temp B-tree sort per request, turning paginated song listings into multi-second queries (reproduced at 5.5s on real hardware; ~90x slower than with correct stats). Every index-creating migration re-triggered the poisoning via the post-migration optimize, and the daily optimizer could re-trigger it on large library changes. Setting analysis_limit on the connection does not help: optimize ignores it. Run a plain full ANALYZE instead: after migrations with schema changes, and in db.Optimize (daily schedule and scan-end). Stats are stored in the database file, so one connection suffices and the per-connection pool loop is gone. The Optimize call at shutdown is removed: stats are maintained at migration/scan/daily points, and an ANALYZE during shutdown only delays it and races container stop timeouts. * perf(db): drop startup PRAGMA optimize that re-poisons planner stats The startup PRAGMA optimize=0x10002 runs SQLite's budget-limited internal ANALYZE (bit 0x02), which writes truncated sqlite_stat1 rows for low-cardinality indexes -- the exact statistics-poisoning this PR set out to eliminate. Because DevOptimizeDB defaults to true, a restart with no pending migrations would re-poison the planner until the next scan or daily Optimize. Remove it: statistics are already refreshed with a full ANALYZE after schema-changing migrations (Init) and via Optimize at scan-end and on the daily schedule, so nothing on the startup path needs to touch them. Also clarify that Optimize is a no-op unless DevOptimizeDB is enabled. * chore(db): remove the DevOptimizeDB flag and skip Optimize on quick scans The flag only gated the optimize/ANALYZE maintenance calls and there is no reason to leave planner statistics unmaintained; the guards are gone along with the flag. The scan-end Optimize now runs only after full scans — quick scans barely move the statistics, and the daily schedule covers drift. * style(scanner): drop redundant comment in runOptimize * chore(persistence): drop the no-op PRAGMA optimize from ScanEnd Mask 0x10000 only selects candidate tables by size change; without the 0x02 action bit optimize does nothing (verified: sqlite_stat1 stays stale after a 100x table growth). The scan-end statistics refresh is db.Optimize's full ANALYZE, and the expression-collation-index concern the old comment guarded against no longer applies. * fix(scanner): run the post-scan ANALYZE in the server process With the external scanner (the default), the scan pipeline runs in a subprocess, so its ANALYZE was invisible to the server: SQLite loads sqlite_stat1 into the process's shared schema cache, and an ANALYZE from another process does not refresh it — verified with the production DSN that even brand-new pool connections keep planning with the old statistics until the server restarts. An in-process ANALYZE, by contrast, is immediately visible to every pooled connection through the same shared cache. Move the full-scan Optimize from the scanner pipeline to the scan controller, which always runs in the server process. * fix(scanner): honor promoted full scans in the optimize gate A quick scan resuming an interrupted full scan is promoted inside the scanner (possibly in a subprocess); mirror the promotion in the controller so the post-scan ANALYZE isn't skipped. * refactor: apply cleanup review findings - drop forceFullRescan's inline ANALYZE: Init already runs a full ANALYZE after any migration batch with schema changes, so upgrades including a full-rescan migration analyzed the whole DB twice - resumingFullScan uses a filtered CountAll instead of fetching and scanning all libraries - document why CallScan (CLI) deliberately skips the post-scan Optimize * perf(db): make planner analysis maintenance resilient Check analysis freshness every 30 minutes and refresh statistics when the last successful run is over 24 hours old or a scan marked them pending. Persist successful analysis state, retry skipped or failed maintenance, coordinate checks with scans, and cover standalone CLI full scans. * perf(db): avoid analyzing routine quick-scan changes Reserve pending analysis for full scans, unscanned libraries, and retry state. Incremental quick scans now rely on the 24-hour freshness window instead of triggering a full ANALYZE at the next maintenance check. * fix(scan): analyze resumed full scans in CLI * fix(db): back off failed analysis retries * feat(db): allow disabling scheduled analysis * test(db): remove redundant analysis coverage * refactor(db): split ANALYZE maintenance into optimize.go and dedupe call sites - move query-planner statistics code from db.go to its own optimize.go (and matching optimize_test.go) - log ANALYZE elapsed time inside Optimize/OptimizeIfNeeded instead of repeating the timing block at every call site - drop the LastDBAnalyzeAttemptAt write on success: it is only read while failures >= 1, and every failure rewrites it first - extract runPostScanAnalysis (cmd) and anyIncludedLibrary (scanner) helpers
116 lines
3.3 KiB
Go
116 lines
3.3 KiB
Go
package migrations
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"fmt"
|
|
"strings"
|
|
"sync"
|
|
|
|
"github.com/navidrome/navidrome/consts"
|
|
)
|
|
|
|
// Use this in migrations that need to communicate something important (breaking changes, forced reindexes, etc...)
|
|
func notice(ctx context.Context, tx *sql.Tx, msg string) {
|
|
if isDBInitialized(ctx, tx) {
|
|
line := strings.Repeat("*", len(msg)+8)
|
|
fmt.Printf("\n%s\nNOTICE: %s\n%s\n\n", line, msg, line)
|
|
}
|
|
}
|
|
|
|
// Call this in migrations that requires a full rescan
|
|
func forceFullRescan(ctx context.Context, tx *sql.Tx) error {
|
|
_, err := tx.ExecContext(ctx, fmt.Sprintf(`
|
|
INSERT OR REPLACE into property (id, value) values ('%s', '1');
|
|
`, consts.FullScanAfterMigrationFlagKey))
|
|
return err
|
|
}
|
|
|
|
// sq := Update(r.tableName).
|
|
// Set("last_scan_started_at", time.Now()).
|
|
// Set("full_scan_in_progress", fullScan).
|
|
// Where(Eq{"id": id})
|
|
|
|
var (
|
|
once sync.Once
|
|
initialized bool
|
|
)
|
|
|
|
func isDBInitialized(ctx context.Context, tx *sql.Tx) bool {
|
|
once.Do(func() {
|
|
rows, err := tx.QueryContext(ctx, "select count(*) from property where id=?", consts.InitialSetupFlagKey)
|
|
checkErr(err)
|
|
initialized = checkCount(rows) > 0
|
|
})
|
|
return initialized
|
|
}
|
|
|
|
func checkCount(rows *sql.Rows) (count int) {
|
|
for rows.Next() {
|
|
err := rows.Scan(&count)
|
|
checkErr(err)
|
|
}
|
|
return count
|
|
}
|
|
|
|
func checkErr(err error) {
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
}
|
|
|
|
type (
|
|
execFunc func() error
|
|
execStmtFunc func(stmt string) execFunc
|
|
addColumnFunc func(tableName, columnName, columnType, defaultValue, initialValue string) execFunc
|
|
)
|
|
|
|
func createExecuteFunc(ctx context.Context, tx *sql.Tx) execStmtFunc {
|
|
return func(stmt string) execFunc {
|
|
return func() error {
|
|
_, err := tx.ExecContext(ctx, stmt)
|
|
return err
|
|
}
|
|
}
|
|
}
|
|
|
|
// Hack way to add a new `not null` column to a table, setting the initial value for existing rows based on a
|
|
// SQL expression. It is done in 3 steps:
|
|
// 1. Add the column as nullable. Due to the way SQLite manipulates the DDL in memory, we need to add extra padding
|
|
// to the default value to avoid truncating it when changing the column to not null
|
|
// 2. Update the column with the initial value
|
|
// 3. Change the column to not null with the default value
|
|
//
|
|
// Based on https://stackoverflow.com/a/25917323
|
|
func createAddColumnFunc(ctx context.Context, tx *sql.Tx) addColumnFunc {
|
|
return func(tableName, columnName, columnType, defaultValue, initialValue string) execFunc {
|
|
return func() error {
|
|
// Format the `default null` value to have the same length as the final defaultValue
|
|
finalLen := len(fmt.Sprintf(`%s not`, defaultValue))
|
|
tempDefault := fmt.Sprintf(`default %s null`, strings.Repeat(" ", finalLen))
|
|
_, err := tx.ExecContext(ctx, fmt.Sprintf(`
|
|
alter table %s add column %s %s %s;`, tableName, columnName, columnType, tempDefault))
|
|
if err != nil {
|
|
return err
|
|
}
|
|
_, err = tx.ExecContext(ctx, fmt.Sprintf(`
|
|
update %s set %s = %s where %[2]s is null;`, tableName, columnName, initialValue))
|
|
if err != nil {
|
|
return err
|
|
}
|
|
_, err = tx.ExecContext(ctx, fmt.Sprintf(`
|
|
PRAGMA writable_schema = on;
|
|
UPDATE sqlite_master
|
|
SET sql = replace(sql, '%[1]s %[2]s %[5]s', '%[1]s %[2]s default %[3]s not null')
|
|
WHERE type = 'table'
|
|
AND name = '%[4]s';
|
|
PRAGMA writable_schema = off;
|
|
`, columnName, columnType, defaultValue, tableName, tempDefault))
|
|
if err != nil {
|
|
return err
|
|
}
|
|
return err
|
|
}
|
|
}
|
|
}
|