b96e359b4e
* feat(catalog): typo-tolerant search for the Postgres (non-Meilisearch) path
## What this does (plain language)
When someone searches the library and misspells a title — "intersteller",
"godfathr", "jurasic" — the Postgres-backed search used to return nothing,
because it only did exact full-text matching. This adds a "did you mean"
fallback: when the normal search finds little or nothing, we run a second,
typo-tolerant lookup and surface the closest titles.
This only affects deployments that search via Postgres (the fallback path).
Meilisearch already does its own typo tolerance and is left untouched.
## Why not just make the main query fuzzy
The obvious approach — OR a trigram similarity match into the main search — is a
performance trap. The trigram operator is "lossy", so Postgres re-checks every
near-miss candidate by rebuilding three title search-vectors per row. On a real
library that turned routine searches into multi-second queries.
Measured on a 175k-title dev database:
- exact full-text only: ~60 ms
- fuzzy OR'd into the main query: ~217 ms (and far worse on prod-sized data)
## How it works
The fuzzy arm is a completely separate query (buildFuzzySearchSQL). It matches
only on the trigram-indexed title_normalized column and ranks only by
similarity() on that same column — it never touches the title search-vectors, so
it pays no per-row rebuild. It runs only when the exact search is "sparse" (fewer
than 5 hits) and the query is long enough for the trigram index to help (>= 4
characters), so the common case stays on the fast exact path. It is wired into
SearchPage (not just the thin Search wrapper) so the catalog search provider
benefits too.
## Measured on the live 183k-title catalog (read-only EXPLAIN ANALYZE)
- exact query for a typo: ~0.8 ms (0 hits -> triggers the fallback)
- fuzzy fallback query: ~5-27 ms, always via the trigram index, with no
search-vector rebuild
- "intersteller" -> Interstellar (similarity 0.63)
- "godfathr" -> GodFather (0.58), The Godfather
- "breakin" (134 exact hits) -> fuzzy correctly does NOT fire
Shared scope predicates (type / library / access / manga-exclusion) are extracted
into appendSearchScopeFilters so the exact and fuzzy queries filter identically.
Adapted from the earlier feat/search-fuzzy-fallback prototype onto main's current
SearchPage / includeTotal architecture.
* refactor(catalog): correct fuzzy-search pagination and parse the query once
Follow-up to the fuzzy fallback, from an adversarial code review. Two things: a
pagination correctness fix and a small performance/readability cleanup. Both were
validated against the live 183k-title catalog.
## The pagination bug (plain language)
Fuzzy results are shown after the exact results, as one combined list. The first
version stitched that list together with page-offset math, and got the math wrong
past the first page:
- the reported result count grew as you paged (page 1 said "31 results",
page 2 said "33");
- titles shown on page 1 could reappear on page 2;
- paging far past the end still ran the (pointless) fuzzy query every time;
- a tiny page size (e.g. an autocomplete asking for 3) could hide the fuzzy
results behind a page the client was told did not exist.
## The fix
Because the fuzzy fallback only runs when exact results are sparse (< 5) and the
fuzzy part is capped at 50, the whole combined list is tiny. So instead of
fragile per-page offset math, we now fetch that small combined list once and take
the requested slice in memory. Every page is then correct by construction: stable
total, no repeats, no wasted work past the end.
Before -> after, typo search "intersteller" (21 results, page size 5):
- total reported on page 2: 31 then 33 (drifting) -> 21 (stable)
- repeated titles across pages: yes -> none
- request past the end (offset 500): 2-3 DB queries -> 0 extra queries
- autocomplete (page size 1): fuzzy hidden -> paginates correctly
Cursor-style callers (that don't ask for a total) can't locate the boundary
between the two blocks on a later page, so they now get the fuzzy results as a
single terminal first page — no misleading "more results" flag.
## The cleanup
The raw query string was being parsed three times per search (once for the
eligibility check, once in each SQL builder). It is now parsed once in SearchPage
and passed down; the shared search-text derivation is extracted so the two
builders can't disagree; and the normalized form the eligibility gate needs is
precomputed at parse time. ("Performance first", per the repo guidelines.)
Also considered and rejected: excluding exact hits from the fuzzy query with a
NOT(full-text) clause instead of by id. It reintroduced the search-vector rebuild
the whole design avoids — measured ~51 ms vs ~20 ms on the worst case — so
id-based exclusion stayed.
Known limitation: the fuzzy path re-reads the small exact block in a second
query, so a title written in the sub-millisecond gap between the two reads could
be missed until the next search. Harmless and inherent to a multi-query design.
* fix(catalog): close fuzzy-search library-scope leak and restore small-limit cursor recall
Addresses two findings from the PR #386 review bots.
## Library-scope leak (Codex P1)
The search scope helper shared by the FTS query and the trigram fuzzy fallback
filtered libraries with `JOIN media_item_libraries mil` +
`NOT (mil.media_folder_id = ANY($disabled))`. An item linked to BOTH a disabled
and a non-disabled library fans out to two joined rows; the non-disabled row
satisfies the deny check, GROUP BY collapses the item back, and it surfaces in
search results despite the disabled library. Because the new fuzzy fallback
reuses this helper, typo searches could leak disabled-library items too.
appendSearchScopeFilters now delegates to the leak-safe
appendLibraryAccessConditions (access_filter.go), which emits item-scoped
EXISTS/NOT EXISTS subqueries — the same form GetByIDs/EnsureAccessible already
use — and needs no membership JOIN. The disabled-only path keeps its
argument-free positive-membership EXISTS so orphan items don't slip through a
vacuous NOT EXISTS. The scored CTEs keep GROUP BY (now required only for the
MAX() ranking aggregates). New regression test pins the EXISTS/NOT EXISTS shape
and the absence of a JOIN for both the FTS and fuzzy builders.
## Small-limit cursor recall (Codex P2)
In cursor mode (include_total=false) the FTS probe fetched only limit+1 rows.
For a tiny caller limit (e.g. an autocomplete asking for 2) with a few incidental
exact hits, that made ftsHasMore true, so the block never looked "sparse" and the
typo fallback never fired — and subsequent offsets are barred from triggering it,
so the fuzzy results were unreachable entirely.
SearchPage now floors the cursor-mode probe at fuzzyFallbackThreshold rows, and
execSearchBlock returns the pre-trim row count so sparsity is judged as
`fetched < threshold` independent of the caller's page size. The returned page is
still trimmed to limit with correct hasMore. Exact mode is unchanged (it judges
sparsity by the page-independent window count).
* fix(catalog): harden fuzzy-search fallback per adversarial review
Addresses the confirmed findings from a deep review of the fuzzy-search
fallback:
- Cursor mode now enters the fallback only when the whole sparse FTS
block fits the caller's page, so the terminal fuzzy page can never
hide exact matches the plain hasMore path would have surfaced
(jellycompat clients with EnableTotalRecordCount=false lost matches).
- execSearchBlock takes a querier and returns its untrimmed rows;
SearchPage hands the already-fetched block to the fallback instead of
re-running an identical FTS query on every sparse search.
- Fuzzy truncation is detected with LIMIT cap+1 instead of a
COUNT(*) OVER () window count that only fed a debug log; truncated
exact-mode responses now report total_exact=false rather than
presenting the cap as an exact count.
- The fuzzy query runs in a transaction pinning
pg_trgm.similarity_threshold via SET LOCAL, so match quality cannot
drift with cluster configuration.
- When the FTS block has real hits, fuzzy augmentation demands
similarity >= 0.45 so correctly-spelled sparse queries only gain
near-identical titles instead of base-threshold trigram noise.
- filterCatalogSearchItems no longer erases fuzzy matches on the
filtered/sorted/prefix resolver path: a typo token is never a
substring of the titles it matched, which left typo search returning
zero results there while the plain search box showed matches.
- The cursor probe floor applies only when the fallback can fire;
cursor fuzzy fetches no more rows than the terminal page can serve.
- slog.Debug -> slog.DebugContext (sloglint); reuse
contentIDsFromMediaItems instead of a duplicate helper; document the
title-only fuzzy scope.
Verified against the dev deployment: stable totals across pages, no
duplicates, small-limit cursor recall restored, filtered-path typo
search working, ~160ms typo-path latency.
Part of PR #386.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(catalog): reach long titles via strict word similarity in fuzzy search
Full-string trigram similarity is diluted by every extra trigram a long
title contributes, so a typo of one word could never reach titles like
"Avengers: Endgame" ("avegners" scores ~0.38 against "avengers" but far
below threshold against the full title). Swap the fuzzy predicate from %
to <<% (strict_word_similarity), which scores the query against the best
word-boundary extent of the title. At equal thresholds <<% is a strict
superset of %, and the existing gin_trgm_ops index serves both — no
migration needed.
The SET LOCAL pin moves to pg_trgm.strict_word_similarity_threshold and
is load-bearing: the 0.6 server default would reject ordinary one-edit
typos outright.
Ranking is strict word similarity first with whole-title similarity()
as tie-break, so near-identical short titles ("The Avengers") sort above
long titles that merely contain the matched word.
The 0.45 augmentation floor deliberately stays on whole-title
similarity(): word similarity rates embedded prefix words far too high
("coral" scores 0.5 against "coraline"), which dev testing showed would
flood a correctly-spelled sparse query with 27 noise rows. Zero-hit
(true typo) queries skip the floor, so the new long-title recall applies
where it matters.
Dev-verified: "avegners" now returns The Avengers first, then Avengers
Grimm / Avengers: Endgame; "coraline" still returns exactly its 4 real
titles; cursor small-limit recall, filtered-path typo search, pagination
stability, and ~160ms typo-path latency all unchanged.
Part of PR #386.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Quick104 <31828688+Quick104@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Silo
Silo is a self-hosted media streaming server for your movies, shows, music, and books. Point it at your media folders and stream to your devices — at home or away — with direct play, remuxing, and hardware-accelerated transcoding handled automatically.
Join the community on Discord. If Silo is useful to you, consider sponsoring the project — see Supporting Silo.
Highlights
- Plays your media, your way — direct play when the device supports it, remux or hardware-accelerated transcode (including NVENC) when it doesn't.
- Web app included — a full-featured web client and admin interface ship with the server.
- Works with apps you already use — optional Jellyfin/Emby-compatible API supports clients such as VidHub, Findroid, and Infuse.
- Household profiles — multiple profiles per account, with per-profile watch state and parental controls.
- Plugin-driven metadata — match and enrich your libraries with providers like TMDB and TVDB, installed as plugins.
- Fast setup — one
docker compose up -dbrings up the whole stack; everything else is configured in the admin UI.
Deploy with Docker (recommended)
The easiest way to run Silo is with Docker Compose. The default stack assumes you do not already have PostgreSQL and Redis available, so it bundles PostgreSQL, Redis, FFmpeg, and the application for a one-command start.
-
Create a
.envfilecp .env.example .env -
Set your media path
Edit
.envand set:MEDIA_ROOT=/path/to/your/mediaMEDIA_ROOTis the one value most users need to change. You can also overrideSILO_DATA_ROOTif you do not want bind mounts under/opt/silo, and change ports if the defaults conflict with something else on the host. -
Start the default integrated stack
docker compose up -dThis starts PostgreSQL, Redis, and the integrated Silo server. The app is available at
http://localhost:8090. Jellyfin-compatible app support is disabled until an administrator enables it in onboarding or admin settings.If you already have PostgreSQL and Redis available, omit those bundled service examples from compose and point Silo at your existing
DATABASE_URLandREDIS_URLinstead.Optional NVIDIA/NVENC
GPU support is kept out of the default compose file so hosts without NVIDIA drivers work unchanged.
Install the NVIDIA Container Toolkit and use a Docker Compose version with GPU reservation support before enabling this override.
Use the optional override file when you want NVENC:
docker compose -f docker-compose.yml -f docker-compose.nvidia.yml up -dIf you want this controlled from
.env, setCOMPOSE_FILE:COMPOSE_FILE=docker-compose.yml:docker-compose.nvidia.yml NVIDIA_GPU_COUNT=1Windows uses
;instead of:between compose files.Then
docker compose up -dwill include the NVIDIA override automatically. -
Configure through the admin UI
Add libraries, users, metadata providers, and playback settings from the web interface.
Bind Mount Layout
The deploy-oriented compose files use host folder mappings rather than Docker-managed volumes.
By default, data is stored under /opt/silo:
/opt/silo/postgres/opt/silo/redis/opt/silo/transcode/opt/silo/catalog-seeds
Media is mounted into the container at /mnt/media from the host path you set in MEDIA_ROOT.
Optional Profiles
The main compose file is integrated-first. These profiles exist for operators testing distributed mode or mirroring a split deployment shape. Most single-host installs should stay on the default integrated service, because it already includes proxying and transcoding.
| Profile | Command | Description |
|---|---|---|
| default | docker compose up -d |
Integrated server plus bundled PostgreSQL and Redis |
proxy |
docker compose --profile proxy up -d |
Start a standalone proxy service for distributed-mode testing |
transcode |
docker compose --profile transcode up -d |
Start a standalone transcode service for distributed-mode testing |
You can enable both optional examples together:
docker compose --profile proxy --profile transcode up -d
If you are splitting workers across multiple hosts, use the separate remote worker example instead of trying to stretch the main compose file across machines.
Advanced Remote Node Example
For a dedicated remote transcode worker, use docker-compose.remote-transcode.yml. That file is intended for a separate worker host that connects back to an existing Silo deployment using shared PostgreSQL and Redis.
Deployment Notes
The default compose stack intentionally bundles PostgreSQL and Redis for ease of setup and assumes a fresh install without those services already available. If you already operate PostgreSQL and Redis, omit those examples from compose and point Silo at your existing infrastructure instead. For serious installs, PostgreSQL is better on a separate VM or a managed service so upgrades, tuning, and backups are isolated from the app host. Redis can stay local for many installs, but externalizing it is also reasonable if you already operate shared infrastructure.
Silo is externally stateful by default rather than fully stateless. Durable application state lives in PostgreSQL. Redis only stores coordination and cache-style data. Silo still writes transient transcode output locally under /tmp/silo-transcode. If you switch userdb.backend=sqlite, Silo also becomes locally stateful at /var/lib/silo/userdb.
Migrating an existing Continuum Docker install should be done with the preflight helper and cutover guide in docs/continuum-to-silo-docker-migration.md.
Configuration
Silo requires only a DATABASE_URL when running from source or against external infrastructure. In the default Docker Compose path, the stack wires the database and Redis URLs for you. All other settings — libraries, metadata providers, transcoding, users — are managed through the admin UI after first launch.
Server Modes
| Mode | Description |
|---|---|
integrated |
Full server: API + frontend + scanner + transcode (default) |
api |
API server only, no local transcoding |
proxy |
Stream proxy node that connects to the shared deployment database and Redis |
transcode |
HLS transcode worker node that connects to the shared deployment database and Redis |
PostgreSQL Auto-Tuning
The default Docker Compose stack does not require a checked-in postgresql.conf.
It enables Silo's pgtune-style OLTP tuning
by default:
POSTGRES_TUNE: auto
When enabled, Silo connects with DATABASE_URL and applies recommendations with
ALTER SYSTEM, which writes to PostgreSQL's postgresql.auto.conf inside the
database data directory. Reloadable settings are applied immediately with
pg_reload_conf(). Settings that PostgreSQL marks as restart-only are written
too, and Silo logs the setting names so you can restart PostgreSQL once:
docker compose restart postgres
The default Compose database user has the required PostgreSQL permissions. If
you use an external PostgreSQL server, make sure the configured DATABASE_URL
user can run ALTER SYSTEM, or set POSTGRES_TUNE=off and manage
PostgreSQL yourself.
For POSTGRES_TUNE_MEMORY=auto, Silo uses the first trustworthy memory source:
a finite Docker cgroup limit, the read-only /host/proc/meminfo mount supplied
by the bundled Compose file, then /proc/meminfo with container safety guards.
Auto-detected memory is treated as a PostgreSQL budget, defaulting to 75% of
detected RAM so Silo, Redis, plugins, transcodes, and the OS retain headroom.
POSTGRES_TUNE_DB_SIZE=auto queries pg_database_size(current_database()) and
classifies the workload by comparing the database size to that memory budget.
Optional tuning overrides:
| Variable | Default | Description |
|---|---|---|
POSTGRES_TUNE_PROFILE |
oltp |
Tuning profile. Only oltp is currently supported. |
POSTGRES_TUNE_MEMORY |
auto |
Server/container RAM, such as 8GB or 32GB; explicit values are used as-is. |
POSTGRES_TUNE_MEMORY_BUDGET_PERCENT |
75 |
Percent of auto-detected RAM used for PostgreSQL recommendations. |
POSTGRES_TUNE_CPUS |
auto |
CPU count used for worker recommendations. |
POSTGRES_TUNE_STORAGE |
ssd |
One of hdd, ssd, san, or nvme. |
POSTGRES_TUNE_DB_SIZE |
auto |
Use less_ram when the database comfortably fits in RAM, mid_ram, or greater_ram for very large databases. |
POSTGRES_TUNE_CONNECTIONS |
100 |
PostgreSQL max_connections; automatically raised if Silo's app pool is configured higher. |
POSTGRES_SHM_SIZE |
8gb |
Docker /dev/shm size for the bundled PostgreSQL container. |
Advanced operators can still supply their own PostgreSQL configuration or
override these env vars. Set POSTGRES_TUNE=off when you do not want Silo to
change PostgreSQL server settings. Settings already written with ALTER SYSTEM
remain in postgresql.auto.conf; reset those PostgreSQL parameters if you later
move fully to a custom postgresql.conf.
Build from Source
If you prefer running Silo without Docker:
-
Install prerequisites: Go 1.24+, Bun 1.0+, PostgreSQL 18+, and FFmpeg.
-
Start PostgreSQL and Redis (skip if you already have them running)
docker compose up -d postgres redisThe main compose file still expects
MEDIA_ROOTto be set even if you only want the bundled PostgreSQL and Redis services, so set that in.envfirst. -
Configure the database connection
cp .env.example .envEdit
.envand setDATABASE_URLto point to your PostgreSQL instance. -
Build and run
make build ./siloThe server starts at
http://localhost:8080by default. All other settings are configured through the admin UI.
Reporting Issues
If you are reporting a bug, install problem, or performance issue, start with the admin workflow and reproduction steps, not Claude/Codex analysis.
Please include:
- What you were trying to do
- Exact steps you took
- What you expected to happen
- What actually happened
- What exact action is slow or broken (
save,scan,browse,import,playback, etc.) - Whether it happens every time or only sometimes
- The library, media type, filter, setting, or value involved
- Version, branch, commit, and deployment details if you know them
- Screenshots, recordings, or log snippets if relevant
If you used Claude/Codex for debugging, put that under Technical notes at the end. Suspected files, SQL output, stack traces, and root-cause theories can be helpful, but only after the workflow and repro steps are clear.
Use this template:
Goal:
Steps:
Expected:
Actual:
What is slow/broken:
Scope:
Version/branch:
Deployment:
Technical notes:
Contributing & Development
Silo is open source and contributions are welcome. See DEVELOPMENT.md for building from source in a dev workflow, running tests, database migrations, and project layout, and CONTRIBUTING.md for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
Supporting Silo
Silo is an open-source hobby project, developed in spare time and funded out of pocket. If you'd like to support development, you can sponsor via GitHub Sponsors.
Donations go directly toward the costs of building and running the project:
- AI development tooling subscriptions (Claude, Codex) used to build and maintain Silo
- Push notification relay infrastructure
- Future development costs
Sponsoring is entirely optional — Silo is and will remain free and open source. Bug reports, contributions, and feedback are just as valuable.
License & Trademarks
Silo's source code is licensed under the GNU Affero General Public License
v3.0 or later (AGPL-3.0-or-later) — see LICENSE.
The Silo name, logo, and wordmark are trademarks of Silo Media L.L.C. and are not covered by the AGPL. You're free to fork and redistribute the code, but forks and redistributions must not use the Silo brand as their identity and must remove or replace the brand assets. Publishing a Silo-branded app to an app store requires written permission. See TRADEMARK.md for what's permitted — including referential use like "compatible with Silo."