navidrome/.github/workflows/pipeline.yml
Deluan Quintão 54de0dbc52
feat(server): implement FTS5-based full-text search (#5079)
* build: add sqlite_fts5 build tag to enable FTS5 support

* feat: add SearchBackend config option (default: fts)

* feat: add buildFTS5Query for safe FTS5 query preprocessing

* feat: add FTS5 search backend with config toggle, refactor legacy search

- Add searchExprFunc type and getSearchExpr() for backend selection
- Rename fullTextExpr to legacySearchExpr
- Add ftsSearchExpr using FTS5 MATCH subquery
- Update fullTextFilter in sql_restful.go to use configured backend

* feat: add FTS5 migration with virtual tables, triggers, and search_participants

Creates FTS5 virtual tables for media_file, album, and artist with
unicode61 tokenizer and diacritic folding. Adds search_participants
column, populates from JSON, and sets up INSERT/UPDATE/DELETE triggers.

* feat: populate search_participants in PostMapArgs for FTS5 indexing

* test: add FTS5 search integration tests

* fix: exclude FTS5 virtual tables from e2e DB restore

The restoreDB function iterates all tables in sqlite_master and
runs DELETE + INSERT to reset state. FTS5 contentless virtual tables
cannot be directly deleted from. Since triggers handle FTS5 sync
automatically, simply skip tables matching *_fts and *_fts_* patterns.

* build: add compile-time guard for sqlite_fts5 build tag

Same pattern as netgo: compilation fails with a clear error if
the sqlite_fts5 build tag is missing.

* build: add sqlite_fts5 tag to reflex dev server config

* build: extract GO_BUILD_TAGS variable in Makefile to avoid duplication

* fix: strip leading * from FTS5 queries to prevent "unknown special query" error

* feat: auto-append prefix wildcard to FTS5 search tokens for broader matching

Every plain search token now gets a trailing * appended (e.g., "love" becomes
"love*"), so searching for "love" also matches "lovelace", "lovely", etc.
Quoted phrases are preserved as exact matches without wildcards. Results are
ordered alphabetically by name/title, so shorter exact matches naturally
appear first.

* fix: clarify comments about FTS5 operator neutralization

The comments said "strip" but the code lowercases operators to
neutralize them (FTS5 operators are case-sensitive). Updated comments
to accurately describe the behavior.

* fix: use fmt.Sprintf for FTS5 phrase placeholders

The previous encoding used rune('0'+index) which silently breaks with
10+ quoted phrases. Use fmt.Sprintf for arbitrary index support.

* fix: validate and normalize SearchBackend config option

Normalize the value to lowercase and fall back to "fts" with a log
warning for unrecognized values. This prevents silent misconfiguration
from typos like "FTS", "Legacy", or "fts5".

* refactor: improve documentation for build tags and FTS5 requirements

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: convert FTS5 query and search backend normalization tests to DescribeTable format

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: add sqlite_fts5 build tag to golangci configuration

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add UISearchDebounceMs configuration option and update related components

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: fall back to legacy search when SearchFullString is enabled

FTS5 is token-based and cannot match substrings within words, so
getSearchExpr now returns legacySearchExpr when SearchFullString
is true, regardless of SearchBackend setting.

* fix: add sqlite_fts5 build tag to CI pipeline and Dockerfile

* fix: add WHEN clauses to FTS5 AFTER UPDATE triggers

Added WHEN clauses to the media_file_fts_au, album_fts_au, and
artist_fts_au triggers so they only fire when FTS-indexed columns
actually change. Previously, every row update (e.g., play count, rating,
starred status) triggered an unnecessary delete+insert cycle in the FTS
shadow tables. The WHEN clauses use IS NOT for NULL-safe comparison of
each indexed column, avoiding FTS index churn for non-indexed updates.

* feat: add SearchBackend configuration option to data and insights components

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: enhance input sanitization for FTS5 by stripping additional punctuation and special characters

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add search_normalized column for punctuated name search (R.E.M., AC/DC)

Add index-time normalization and query-time single-letter collapsing to
fix FTS5 search for punctuated names. A new search_normalized column
stores concatenated forms of punctuated words (e.g., "R.E.M." → "REM",
"AC/DC" → "ACDC") and is indexed in FTS5 tables. At query time, runs of
consecutive single letters (from dot-stripping) are collapsed into OR
expressions like ("R E M" OR REM*) to match both the original tokens and
the normalized form. This enables searching by "R.E.M.", "REM", "AC/DC",
"ACDC", "A-ha", or "Aha" and finding the correct results.

* refactor: simplify isSingleUnicodeLetter to avoid []rune allocation

Use utf8.DecodeRuneInString to check for a single Unicode letter
instead of converting the entire string to a []rune slice.

* feat: define ftsSearchColumns for flexible FTS5 search column inclusion

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: update collapseSingleLetterRuns to return quoted phrases for abbreviations

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: punctuated word handling to improve processing of artist/album names

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add CJK support for search queries with LIKE filters

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: enhance FTS5 search by adding album version support and CJK handling

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: search configuration to use structured options

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: enhance search functionality to support punctuation-only queries and update related tests

Signed-off-by: Deluan <deluan@navidrome.org>

---------

Signed-off-by: Deluan <deluan@navidrome.org>
2026-02-21 17:52:42 -05:00

485 lines
15 KiB
YAML

name: "Pipeline: Test, Lint, Build"
on:
push:
branches:
- master
tags:
- "v*"
pull_request:
branches:
- master
concurrency:
group: ${{ startsWith(github.ref, 'refs/tags/v') && 'tag' || 'branch' }}-${{ github.ref }}
cancel-in-progress: true
env:
CROSS_TAGLIB_VERSION: "2.2.0-1"
CGO_CFLAGS_ALLOW: "--define-prefix"
IS_RELEASE: ${{ startsWith(github.ref, 'refs/tags/') && 'true' || 'false' }}
jobs:
git-version:
name: Get version info
runs-on: ubuntu-latest
outputs:
git_tag: ${{ steps.git-version.outputs.GIT_TAG }}
git_sha: ${{ steps.git-version.outputs.GIT_SHA }}
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
fetch-tags: true
- name: Show git version info
run: |
echo "git describe (dirty): $(git describe --dirty --always --tags)"
echo "git describe --tags: $(git describe --tags `git rev-list --tags --max-count=1`)"
echo "git tag: $(git tag --sort=-committerdate | head -n 1)"
echo "github_ref: $GITHUB_REF"
echo "github_head_sha: ${{ github.event.pull_request.head.sha }}"
git tag -l
- name: Determine git current SHA and latest tag
id: git-version
run: |
GIT_TAG=$(git tag --sort=-committerdate | head -n 1)
if [ -n "$GIT_TAG" ]; then
if [[ "$GITHUB_REF" != refs/tags/* ]]; then
GIT_TAG=${GIT_TAG}-SNAPSHOT
fi
echo "GIT_TAG=$GIT_TAG" >> $GITHUB_OUTPUT
fi
GIT_SHA=$(git rev-parse --short HEAD)
PR_NUM=$(jq --raw-output .pull_request.number "$GITHUB_EVENT_PATH")
if [[ $PR_NUM != "null" ]]; then
GIT_SHA=$(echo "${{ github.event.pull_request.head.sha }}" | cut -c1-8)
GIT_SHA="pr-${PR_NUM}/${GIT_SHA}"
fi
echo "GIT_SHA=$GIT_SHA" >> $GITHUB_OUTPUT
echo "GIT_TAG=$GIT_TAG"
echo "GIT_SHA=$GIT_SHA"
go-lint:
name: Lint Go code
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Download TagLib
uses: ./.github/actions/download-taglib
with:
version: ${{ env.CROSS_TAGLIB_VERSION }}
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: latest
problem-matchers: true
args: --timeout 2m
- name: Run go goimports
run: go run golang.org/x/tools/cmd/goimports@latest -w `find . -name '*.go' | grep -v '_gen.go$' | grep -v '.pb.go$'`
- run: go mod tidy
- name: Verify no changes from goimports and go mod tidy
run: |
git status --porcelain
if [ -n "$(git status --porcelain)" ]; then
echo 'To fix this check, run "make format" and commit the changes'
exit 1
fi
- name: Run go generate
run: go generate ./...
- name: Verify no changes from go generate
run: |
git status --porcelain
if [ -n "$(git status --porcelain)" ]; then
echo 'Generated code is out of date. Run "make gen" and commit the changes'
exit 1
fi
go:
name: Test Go code
runs-on: ubuntu-latest
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v6
- name: Download TagLib
uses: ./.github/actions/download-taglib
with:
version: ${{ env.CROSS_TAGLIB_VERSION }}
- name: Download dependencies
run: go mod download
- name: Test
run: |
pkg-config --define-prefix --cflags --libs taglib # for debugging
go test -shuffle=on -tags netgo,sqlite_fts5 -race ./... -v
- name: Test ndpgen
run: |
cd plugins/cmd/ndpgen
go test -shuffle=on -v
go build -o ndpgen .
./ndpgen --help
js:
name: Test JS code
runs-on: ubuntu-latest
env:
NODE_OPTIONS: "--max_old_space_size=4096"
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
node-version: 24
cache: "npm"
cache-dependency-path: "**/package-lock.json"
- name: npm install dependencies
run: |
cd ui
npm ci
- name: npm lint
run: |
cd ui
npm run check-formatting && npm run lint
- name: npm test
run: |
cd ui
npm test
- name: npm build
run: |
cd ui
npm run build
i18n-lint:
name: Lint i18n files
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- run: |
set -e
for file in resources/i18n/*.json; do
echo "Validating $file"
if ! jq empty "$file" 2>error.log; then
error_message=$(cat error.log)
line_number=$(echo "$error_message" | grep -oP 'line \K[0-9]+')
echo "::error file=$file,line=$line_number::$error_message"
exit 1
fi
done
- run: ./.github/workflows/validate-translations.sh -v
check-push-enabled:
name: Check Docker configuration
runs-on: ubuntu-latest
outputs:
is_enabled: ${{ steps.check.outputs.is_enabled }}
steps:
- name: Check if Docker push is configured
id: check
run: echo "is_enabled=${{ secrets.DOCKER_HUB_USERNAME != '' }}" >> $GITHUB_OUTPUT
build:
name: Build
needs: [js, go, go-lint, i18n-lint, git-version, check-push-enabled]
strategy:
matrix:
platform: [ linux/amd64, linux/arm64, linux/arm/v5, linux/arm/v6, linux/arm/v7, linux/386, linux/riscv64, darwin/amd64, darwin/arm64, windows/amd64, windows/386 ]
runs-on: ubuntu-latest
env:
IS_LINUX: ${{ startsWith(matrix.platform, 'linux/') && 'true' || 'false' }}
IS_ARMV5: ${{ matrix.platform == 'linux/arm/v5' && 'true' || 'false' }}
IS_DOCKER_PUSH_CONFIGURED: ${{ needs.check-push-enabled.outputs.is_enabled == 'true' }}
DOCKER_BUILD_SUMMARY: false
GIT_SHA: ${{ needs.git-version.outputs.git_sha }}
GIT_TAG: ${{ needs.git-version.outputs.git_tag }}
steps:
- name: Sanitize platform name
id: set-platform
run: |
PLATFORM=$(echo ${{ matrix.platform }} | tr '/' '_')
echo "PLATFORM=$PLATFORM" >> $GITHUB_ENV
- uses: actions/checkout@v6
- name: Prepare Docker Buildx
uses: ./.github/actions/prepare-docker
id: docker
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
hub_repository: ${{ vars.DOCKER_HUB_REPO }}
hub_username: ${{ secrets.DOCKER_HUB_USERNAME }}
hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }}
- name: Build Binaries
uses: docker/build-push-action@v6
with:
context: .
file: Dockerfile
platforms: ${{ matrix.platform }}
outputs: |
type=local,dest=./output/${{ env.PLATFORM }}
target: binary
build-args: |
GIT_SHA=${{ env.GIT_SHA }}
GIT_TAG=${{ env.GIT_TAG }}
CROSS_TAGLIB_VERSION=${{ env.CROSS_TAGLIB_VERSION }}
- name: Upload Binaries
uses: actions/upload-artifact@v6
with:
name: navidrome-${{ env.PLATFORM }}
path: ./output
retention-days: 7
- name: Build and push image by digest
id: push-image
if: env.IS_LINUX == 'true' && env.IS_DOCKER_PUSH_CONFIGURED == 'true' && env.IS_ARMV5 == 'false'
uses: docker/build-push-action@v6
with:
context: .
file: Dockerfile
platforms: ${{ matrix.platform }}
labels: ${{ steps.docker.outputs.labels }}
build-args: |
GIT_SHA=${{ env.GIT_SHA }}
GIT_TAG=${{ env.GIT_TAG }}
CROSS_TAGLIB_VERSION=${{ env.CROSS_TAGLIB_VERSION }}
outputs: |
type=image,name=${{ steps.docker.outputs.hub_repository }},push-by-digest=true,name-canonical=true,push=${{ steps.docker.outputs.hub_enabled }}
type=image,name=ghcr.io/${{ github.repository }},push-by-digest=true,name-canonical=true,push=true
- name: Export digest
if: env.IS_LINUX == 'true' && env.IS_DOCKER_PUSH_CONFIGURED == 'true' && env.IS_ARMV5 == 'false'
run: |
mkdir -p /tmp/digests
digest="${{ steps.push-image.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@v6
if: env.IS_LINUX == 'true' && env.IS_DOCKER_PUSH_CONFIGURED == 'true' && env.IS_ARMV5 == 'false'
with:
name: digests-${{ env.PLATFORM }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
push-manifest-ghcr:
name: Push to GHCR
permissions:
contents: read
packages: write
runs-on: ubuntu-latest
needs: [build, check-push-enabled]
if: needs.check-push-enabled.outputs.is_enabled == 'true'
env:
REGISTRY_IMAGE: ghcr.io/${{ github.repository }}
steps:
- uses: actions/checkout@v6
- name: Download digests
uses: actions/download-artifact@v7
with:
path: /tmp/digests
pattern: digests-*
merge-multiple: true
- name: Prepare Docker Buildx
uses: ./.github/actions/prepare-docker
id: docker
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
- name: Create manifest list and push to ghcr.io
working-directory: /tmp/digests
run: |
docker buildx imagetools create $(jq -cr '.tags | map(select(startswith("ghcr.io"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
- name: Inspect image in ghcr.io
run: |
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.docker.outputs.version }}
push-manifest-dockerhub:
name: Push to Docker Hub
runs-on: ubuntu-latest
permissions:
contents: read
needs: [build, check-push-enabled]
if: needs.check-push-enabled.outputs.is_enabled == 'true' && vars.DOCKER_HUB_REPO != ''
continue-on-error: true
steps:
- uses: actions/checkout@v6
- name: Download digests
uses: actions/download-artifact@v7
with:
path: /tmp/digests
pattern: digests-*
merge-multiple: true
- name: Prepare Docker Buildx
uses: ./.github/actions/prepare-docker
id: docker
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
hub_repository: ${{ vars.DOCKER_HUB_REPO }}
hub_username: ${{ secrets.DOCKER_HUB_USERNAME }}
hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }}
- name: Create manifest list and push to Docker Hub
uses: nick-fields/retry@v3
with:
timeout_minutes: 5
max_attempts: 3
retry_wait_seconds: 30
command: |
cd /tmp/digests
docker buildx imagetools create $(jq -cr '.tags | map(select(startswith("ghcr.io") | not)) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf 'ghcr.io/${{ github.repository }}@sha256:%s ' *)
- name: Inspect image in Docker Hub
run: |
docker buildx imagetools inspect ${{ vars.DOCKER_HUB_REPO }}:${{ steps.docker.outputs.version }}
cleanup-digests:
name: Cleanup digest artifacts
runs-on: ubuntu-latest
needs: [push-manifest-ghcr, push-manifest-dockerhub]
if: always() && needs.push-manifest-ghcr.result == 'success'
steps:
- name: Delete unnecessary digest artifacts
env:
GH_TOKEN: ${{ github.token }}
run: |
for artifact in $(gh api repos/${{ github.repository }}/actions/artifacts | jq -r '.artifacts[] | select(.name | startswith("digests-")) | .id'); do
gh api --method DELETE repos/${{ github.repository }}/actions/artifacts/$artifact
done
msi:
name: Build Windows installers
needs: [build, git-version]
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
- uses: actions/download-artifact@v7
with:
path: ./binaries
pattern: navidrome-windows*
merge-multiple: true
- name: Install Wix
run: sudo apt-get install -y wixl jq
- name: Build MSI
env:
GIT_TAG: ${{ needs.git-version.outputs.git_tag }}
run: |
rm -rf binaries/msi
sudo GIT_TAG=$GIT_TAG release/wix/build_msi.sh ${GITHUB_WORKSPACE} 386
sudo GIT_TAG=$GIT_TAG release/wix/build_msi.sh ${GITHUB_WORKSPACE} amd64
du -h binaries/msi/*.msi
- name: Upload MSI files
uses: actions/upload-artifact@v6
with:
name: navidrome-windows-installers
path: binaries/msi/*.msi
retention-days: 7
release:
name: Package/Release
needs: [build, msi]
runs-on: ubuntu-latest
outputs:
package_list: ${{ steps.set-package-list.outputs.package_list }}
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
fetch-tags: true
- uses: actions/download-artifact@v7
with:
path: ./binaries
pattern: navidrome-*
merge-multiple: true
- run: ls -lR ./binaries
- name: Set RELEASE_FLAGS for snapshot releases
if: env.IS_RELEASE == 'false'
run: echo 'RELEASE_FLAGS=--skip=publish --snapshot' >> $GITHUB_ENV
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6
with:
version: '~> v2'
args: "release --clean -f release/goreleaser.yml ${{ env.RELEASE_FLAGS }}"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Remove build artifacts
run: |
ls -l ./dist
rm ./dist/*.tar.gz ./dist/*.zip
- name: Upload all-packages artifact
uses: actions/upload-artifact@v6
with:
name: packages
path: dist/navidrome_0*
- id: set-package-list
name: Export list of generated packages
run: |
cd dist
set +x
ITEMS=$(ls navidrome_0* | sed 's/^navidrome_0[^_]*_linux_//' | jq -R -s -c 'split("\n")[:-1]')
echo $ITEMS
echo "package_list=${ITEMS}" >> $GITHUB_OUTPUT
upload-packages:
name: Upload Linux PKG
runs-on: ubuntu-latest
needs: [release]
strategy:
matrix:
item: ${{ fromJson(needs.release.outputs.package_list) }}
steps:
- name: Download all-packages artifact
uses: actions/download-artifact@v7
with:
name: packages
path: ./dist
- name: Upload all-packages artifact
uses: actions/upload-artifact@v6
with:
name: navidrome_linux_${{ matrix.item }}
path: dist/navidrome_0*_linux_${{ matrix.item }}
# delete-artifacts:
# name: Delete unused artifacts
# runs-on: ubuntu-latest
# needs: [upload-packages]
# steps:
# - name: Delete all-packages artifact
# env:
# GH_TOKEN: ${{ github.token }}
# run: |
# for artifact in $(gh api repos/${{ github.repository }}/actions/artifacts | jq -r '.artifacts[] | select(.name | startswith("packages")) | .id'); do
# gh api --method DELETE repos/${{ github.repository }}/actions/artifacts/$artifact
# done