QuickandGitHub
10394b0a05
* fix(scanner): stop hiding media when a library root is offline
A scan that cannot read a library root found no files there, so every
cataloged file under it was marked missing. Catalog reads all filter on
missing_since IS NULL, so marking is equivalent to deletion from a user's
point of view: the title leaves browse, search and next-up, and playback
answers "Source media file is missing" for media that is intact on disk.
The dead-root protection already existed but only guarded the destructive
operations. protectedConfiguredRoots was computed *after* the marking loop
in scanPaths, and applyScopedScan received the protected set but applied it
only to its force-delete branch. So an unreachable root could not lose its
rows, but could still have its entire catalog hidden until the next
successful scan.
On a CephFS deployment whose per-library subvolume mounts flap, this marked
190 present files missing in a single day — 15% of all missing-flagged rows
were files sitting untouched on disk, some flagged more than 20 hours after
their last write.
Hoist the probe above the marking loop and skip files under an unreachable
or suspect-empty root in both the folder and scoped paths. Pass the
unreachable set to the walked-scope call too: a nested child mount can die
under a healthy parent, and its rows are inside the parent's scope.
An offline root tells us nothing about whether its files exist. The only
safe reading is to leave them alone and let the next good scan decide.
Genuine deletions under a reachable root are unaffected and still marked
and swept on the same schedule as before.
Report the count as ScanResult.MissingSkippedProtected and log it, so an
operator can tell "my library shrank" from "my mount dropped".
Known gap: a suspect-empty *nested child* root under a healthy parent is
still marked missing, because the suspect set is not resolved until after
the walk loop. Unreachable roots — the case observed in production — are
covered.
* fix(scanner): protect suspect-empty and partially-walked roots too
Addresses review findings on #472. The original change guarded missing-marking
against probe-unreachable roots, but left three ways for a storage fault to
still hide a healthy library.
Suspect-empty detection was reactive. suspectEmptyRoots asked
ListRootsWithOnlyMissingFiles, which returns a root only once it has NO live
rows left. On the first scan after a mount drops — the moment that matters —
the rows are still live, so the root was not classified suspect and the scan
marked everything missing. The protection then engaged on the next scan, in
time to protect the wreckage. Ask ListRootsWithCatalogedFiles instead: any
cataloged row under an empty-but-reachable root is the lost-mount signature.
Intentional emptying is still reachable through the operator's one-time
cleanup allowance, which is the deliberate path for it.
Nested suspect-empty children were unprotected. Root compaction sends only the
populated parent through the walked-scope branch, which received only
unreachableRoots, so an empty child mountpoint had its rows marked missing on
its parent scanning cleanly. Pass the suspect set as well.
Partial walks were treated as authoritative. walkLogicalTree deliberately
swallows per-entry Lstat/ReadDir failures so one bad file cannot abort a scan
of a million, and collectLogicalFilePaths passed nil for the failure counter —
so the video path had no signal at all. A mount dying partway through
traversal produced a short file list indistinguishable from a large deletion.
Thread the counter through, and exclude a scope whose walk came back
incomplete from missing reconciliation, mirroring what the ebook scanner
already does via ebookRootScan.failed.
Also extract the duplicated mark-missing loop into markMissingExcludingProtected
so the folder and scoped paths cannot drift, and correct two comments that
still described the pre-fix "files are marked missing" behaviour — the exact
text a future reader would have trusted when reintroducing this bug.
TestScanFolderNestedSuspectEmptyChildRootProtection asserted the old
behaviour and is updated accordingly.
* fix(scanner): scope walk-failure protection and stop pruning on partial walks
Addresses the second Codex review round on #472. The previous commit's
incomplete-walk protection was too blunt in one direction and applied too late
in another.
Walk failures were counted, not located, and any non-zero count protected the
whole library root. A dangling symlink is both common and permanent, so that
would have suppressed missing-file reconciliation for its entire root on every
future scan — genuinely deleted titles would stay live indefinitely. That is
the same class of bug as the one this PR fixes, pointing the other way.
recordWalkFailure now records the logical path of each unreadable entry, and
only those paths are protected. Per-entry failures record the child path, so a
dangling symlink protects itself and nothing else, while a directory that
cannot be read protects its subtree.
Snapshot and group pruning ran before the protection. reconcileScannedRoots
and reconcileScannedGroups delete whatever the walk did not see, and both run
ahead of the missing-file guard, so a partial walk still dropped root
snapshots, observed locations and group locations for the unread portion —
corrupting later metadata matching even though the media_files rows survived.
Upserting what was seen is always safe; pruning now waits for a scan that read
the whole tree.
The confirmed-cleanup allowance was consumed to no effect for nested suspect
children. The walked-parent branch protected them unconditionally and runs
before the allowance is consumed, and an already-reconciled scope cannot be
revisited — so arming the allowance burned the confirmation while the child's
rows stayed live forever. Read the allowance without consuming it before the
walk loop, and honour it there. Unreachable roots stay protected either way:
an outage is never a confirmation to erase a catalog.
Two new regression tests, plus signature updates in the ebook pipeline, which
already tracked walk failures and now shares the path-based representation.
* fix(scanner): re-probe nested roots and gate group pruning on walk completeness
Third Codex review round on #472; both findings confirmed.
Group pruning ignored walk completeness in the subtree path. scanPaths passed
the completeness decision to reconcileScannedRoots but left
reconcileScannedGroups on !allowEmptyRootGuard, which is always true for
ScanSubtree — so a subtree scan that hit an unreadable directory still replaced
group snapshots and locations from a partial inventory. Same rule now applies
to both.
Nested roots were not re-probed before their parent was reconciled. Root
compaction folds a child mount into its parent for traversal, so a child that
is healthy at the initial probe but drops before the parent is walked leaves no
scope of its own, and the post-walk re-probe only revisits scopes that walked
empty. The parent walks files, looks healthy, and the child's rows are marked
missing on its success. reprobeNestedRoots re-checks this root's configured
children immediately before reconciling, protecting any that have since become
unreachable — or suspect-empty, unless the operator has confirmed cleanup.
Also guard suspectEmptyRoots against a nil file repository, matching
emptyCleanupArmed: without a catalog there is nothing to protect.
* fix(scanner): keep re-probed outages protected through folder-wide cleanup
Fourth Codex review round on #472; both findings confirmed. The first could
destroy data.
reprobeNestedRoots protected a root it found offline only for the scope being
reconciled, then discarded the result. The folder-wide membership reconcile and
the trash sweep afterwards rebuilt their protected set from the initial probe
alone, so rows under a child that dropped mid-scan — already marked missing and
past the removal grace — were hard-deleted by the very scan that noticed the
outage. Accumulate those roots in reprobedRoots, fold them into
protectedScanRoots, and reuse that set for the membership reconcile and sweep
instead of rebuilding. They now also land in ScanResult.UnreachableRoots so the
folder warning reflects the outage rather than presenting a partial scan as
clean.
Snapshot and group pruning was enabled for scopes that were never walked. The
gate was len(walkFailures) == 0, but an unreachable root gets nil walkRoots, so
it has no walk and therefore no failures — and pruning then deleted its
snapshots, observed locations and group locations even though its media rows
were protected. The same held for a suspect-empty child compacted into a
populated parent. Pruning now additionally requires that the scope was actually
walked and contains no protected path.
The new test pins that the sweep honours the protected set it is given. It does
not reproduce the mid-scan race itself: staging that needs the drop to land
between the probe and the walk, which a test cannot reach without hooks. That
path is covered by inspection, and the test comment says so rather than
implying coverage it does not have.
* fix(scanner): route every protection source through one folder-wide set
Fifth Codex review round on #472. Two P1s, one of them the second data-loss
path in this area — and the direct sibling of the one fixed in 35326adc, which
is the reason this commit changes the structure rather than patching another
edge.
Rows beneath a directory the walk could not read were protected only inside
applyScopedScan. The folder-wide protected set was rebuilt from the probe
results alone, so DeleteMissingByFolder could permanently delete rows past the
removal grace under a subtree this scan never managed to read — deleting on the
strength of an observation that was never made.
The recurring defect is structural: protection is discovered in several places
(initial probe, mid-loop re-probe, per-scope walk failures) and consumed in
several more (scoped reconcile, membership reconcile, trash sweep), and each
fix so far has wired up one edge and missed another. Every source now
accumulates folder-wide and every consumer reads the combined set, so a new
source has one place to register instead of several to remember.
reprobeNestedRoots classified from two probe batches. It called
probeUnreachableRoots, then suspectEmptyRoots probed the same paths again; a
child dropping between the samples was reachable to the first and discarded by
the second, which only returns reachable-and-empty roots. It now classifies
both states from one batch, so the disconnect it exists to catch cannot fall
between its own probes.
Re-probed roots kept their classification instead of being collapsed into
unreachableRoots, which had been reporting a suspect-empty child as
unreachable and giving operators contradictory failure information.
The new regression test is verified to fail with the propagation disabled and
pass with it, rather than assumed to cover the path.
Not addressed: the cleanup allowance is read without being reserved, so two
overlapping full scans of one folder can both observe it armed. Narrow, needs
a transactional reserve in the scan-claim query, and is left for follow-up
rather than bundled here.
* fix(scanner): resolve root protection before scoped metadata pruning
Sixth Codex review round on #472.
scanPaths pruned before it knew what was protected. reconcileScannedRoots and
reconcileScannedGroups ran roughly 160 lines ahead of protectedConfiguredRoots,
so a ScanSubtree of a mount that dropped but left a reachable empty mountpoint
walked clean, reported no failures, and pruned root snapshots and observed and
group locations against that empty inventory — preserving the media rows while
deleting the metadata describing them. Protection is now resolved before any
reconciliation, and both prunes share one decision, matching applyScopedScan.
Pending empty scopes never re-probed their nested children. A parent whose only
media lives in a child walks empty when that child drops, so it lands in
pendingEmptyScopes rather than the populated-scope branch where
reprobeNestedRoots ran. Probing the parent alone proves nothing: it still holds
the child's bare mountpoint directory, so it reads present and non-empty. With
a healthy sibling keeping the folder-wide empty guard quiet, nothing protected
the child. Both branches now re-probe.
MissingSkippedProtected never left the scanner. Both ingest-to-result
conversions copied every other cleanup count but not this one, and
events.ScanRunResult had no field, so scan history, completion events and API
responses reported an all-zero no-op for a scan that skipped files because
storage was offline. Added as a new field, which is additive under the v1 API
rules.
Test honesty: the new test does NOT exercise the pending-scope re-probe. It
empties the child before the scan, so the initial probe classifies it and
protection arrives by that path — verified by confirming the test still passes
with the re-probe disabled. It is named and commented for what it does cover.
The mid-scan race behind both re-probe fixes needs the drop to land between the
probe and the walk, which is not reachable from a test without hooks; those
fixes rest on inspection.
Silo
Silo is a self-hosted media streaming server for your movies, shows, music, and books. Point it at your media folders and stream to your devices — at home or away — with direct play, remuxing, and hardware-accelerated transcoding handled automatically.
Join the community on Discord. If Silo is useful to you, consider sponsoring the project — see Supporting Silo.
Highlights
- Plays your media, your way — direct play when the device supports it, remux or hardware-accelerated transcode (including NVENC) when it doesn't.
- Web app included — a full-featured web client and admin interface ship with the server.
- Works with apps you already use — optional Jellyfin/Emby-compatible API supports clients such as VidHub, Findroid, and Infuse.
- Household profiles — multiple profiles per account, with per-profile watch state and parental controls.
- Plugin-driven metadata — match and enrich your libraries with providers like TMDB and TVDB, installed as plugins.
- Fast setup — one
docker compose up -dbrings up the whole stack; everything else is configured in the admin UI.
Deploy with Docker (recommended)
The easiest way to run Silo is with Docker Compose. The default stack assumes you do not already have PostgreSQL and Redis available, so it bundles PostgreSQL, Redis, FFmpeg, and the application for a one-command start.
-
Create a
.envfilecp .env.example .env -
Set your media path
Edit
.envand set:MEDIA_ROOT=/path/to/your/mediaMEDIA_ROOTis the one value most users need to change. You can also overrideSILO_DATA_ROOTif you do not want bind mounts under/opt/silo, and change ports if the defaults conflict with something else on the host. -
Start the default integrated stack
docker compose up -dThis starts PostgreSQL, Redis, and the integrated Silo server. The app is available at
http://localhost:8090. Jellyfin-compatible app support is disabled until an administrator enables it in onboarding or admin settings.If you already have PostgreSQL and Redis available, omit those bundled service examples from compose and point Silo at your existing
DATABASE_URLandREDIS_URLinstead.Optional NVIDIA/NVENC
GPU support is kept out of the default compose file so hosts without NVIDIA drivers work unchanged.
Install the NVIDIA Container Toolkit and use a Docker Compose version with GPU reservation support before enabling this override.
Use the optional override file when you want NVENC:
docker compose -f docker-compose.yml -f docker-compose.nvidia.yml up -dIf you want this controlled from
.env, setCOMPOSE_FILE:COMPOSE_FILE=docker-compose.yml:docker-compose.nvidia.yml NVIDIA_GPU_COUNT=1Windows uses
;instead of:between compose files.Then
docker compose up -dwill include the NVIDIA override automatically. -
Configure through the admin UI
Add libraries, users, metadata providers, and playback settings from the web interface.
Bind Mount Layout
The deploy-oriented compose files use host folder mappings rather than Docker-managed volumes.
By default, data is stored under /opt/silo:
/opt/silo/postgres/opt/silo/redis/opt/silo/transcode/opt/silo/catalog-seeds
Media is mounted into the container at /mnt/media from the host path you set in MEDIA_ROOT.
Optional Profiles
The main compose file is integrated-first. These profiles exist for operators testing distributed mode or mirroring a split deployment shape. Most single-host installs should stay on the default integrated service, because it already includes proxying and transcoding.
| Profile | Command | Description |
|---|---|---|
| default | docker compose up -d |
Integrated server plus bundled PostgreSQL and Redis |
proxy |
docker compose --profile proxy up -d |
Start a standalone proxy service for distributed-mode testing |
transcode |
docker compose --profile transcode up -d |
Start a standalone transcode service for distributed-mode testing |
You can enable both optional examples together:
docker compose --profile proxy --profile transcode up -d
If you are splitting workers across multiple hosts, use the separate remote worker example instead of trying to stretch the main compose file across machines.
Advanced Remote Node Example
For a dedicated remote transcode worker, use docker-compose.remote-transcode.yml. That file is intended for a separate worker host that connects back to an existing Silo deployment using shared PostgreSQL and Redis.
Deployment Notes
The default compose stack intentionally bundles PostgreSQL and Redis for ease of setup and assumes a fresh install without those services already available. If you already operate PostgreSQL and Redis, omit those examples from compose and point Silo at your existing infrastructure instead. For serious installs, PostgreSQL is better on a separate VM or a managed service so upgrades, tuning, and backups are isolated from the app host. Redis can stay local for many installs, but externalizing it is also reasonable if you already operate shared infrastructure.
Silo is externally stateful by default rather than fully stateless. Durable application state lives in PostgreSQL. Redis only stores coordination and cache-style data. Silo still writes transient transcode output locally under /tmp/silo-transcode. If you switch userdb.backend=sqlite, Silo also becomes locally stateful at /var/lib/silo/userdb.
Migrating an existing Continuum Docker install should be done with the preflight helper and cutover guide in docs/continuum-to-silo-docker-migration.md.
Configuration
Silo requires only a DATABASE_URL when running from source or against external infrastructure. In the default Docker Compose path, the stack wires the database and Redis URLs for you. All other settings — libraries, metadata providers, transcoding, users — are managed through the admin UI after first launch.
Server Modes
| Mode | Description |
|---|---|
integrated |
Full server: API + frontend + scanner + transcode (default) |
api |
API server only, no local transcoding |
proxy |
Stream proxy node that connects to the shared deployment database and Redis |
transcode |
HLS transcode worker node that connects to the shared deployment database and Redis |
PostgreSQL Auto-Tuning
The default Docker Compose stack does not require a checked-in postgresql.conf.
It enables Silo's pgtune-style OLTP tuning
by default:
POSTGRES_TUNE: auto
When enabled, Silo connects with DATABASE_URL and applies recommendations with
ALTER SYSTEM, which writes to PostgreSQL's postgresql.auto.conf inside the
database data directory. Reloadable settings are applied immediately with
pg_reload_conf(). Settings that PostgreSQL marks as restart-only are written
too, and Silo logs the setting names so you can restart PostgreSQL once:
docker compose restart postgres
The default Compose database user has the required PostgreSQL permissions. If
you use an external PostgreSQL server, make sure the configured DATABASE_URL
user can run ALTER SYSTEM, or set POSTGRES_TUNE=off and manage
PostgreSQL yourself.
For POSTGRES_TUNE_MEMORY=auto, Silo uses the first trustworthy memory source:
a finite Docker cgroup limit, the read-only /host/proc/meminfo mount supplied
by the bundled Compose file, then /proc/meminfo with container safety guards.
Auto-detected memory is treated as a PostgreSQL budget, defaulting to 75% of
detected RAM so Silo, Redis, plugins, transcodes, and the OS retain headroom.
POSTGRES_TUNE_DB_SIZE=auto queries pg_database_size(current_database()) and
classifies the workload by comparing the database size to that memory budget.
Optional tuning overrides:
| Variable | Default | Description |
|---|---|---|
POSTGRES_TUNE_PROFILE |
oltp |
Tuning profile. Only oltp is currently supported. |
POSTGRES_TUNE_MEMORY |
auto |
Server/container RAM, such as 8GB or 32GB; explicit values are used as-is. |
POSTGRES_TUNE_MEMORY_BUDGET_PERCENT |
75 |
Percent of auto-detected RAM used for PostgreSQL recommendations. |
POSTGRES_TUNE_CPUS |
auto |
CPU count used for worker recommendations. |
POSTGRES_TUNE_STORAGE |
ssd |
One of hdd, ssd, san, or nvme. |
POSTGRES_TUNE_DB_SIZE |
auto |
Use less_ram when the database comfortably fits in RAM, mid_ram, or greater_ram for very large databases. |
POSTGRES_TUNE_CONNECTIONS |
100 |
PostgreSQL max_connections; automatically raised if Silo's app pool is configured higher. |
POSTGRES_SHM_SIZE |
8gb |
Docker /dev/shm size for the bundled PostgreSQL container. |
Advanced operators can still supply their own PostgreSQL configuration or
override these env vars. Set POSTGRES_TUNE=off when you do not want Silo to
change PostgreSQL server settings. Settings already written with ALTER SYSTEM
remain in postgresql.auto.conf; reset those PostgreSQL parameters if you later
move fully to a custom postgresql.conf.
Build from Source
If you prefer running Silo without Docker:
-
Install prerequisites: Go 1.24+, Bun 1.0+, PostgreSQL 18+, and FFmpeg.
-
Start PostgreSQL and Redis (skip if you already have them running)
docker compose up -d postgres redisThe main compose file still expects
MEDIA_ROOTto be set even if you only want the bundled PostgreSQL and Redis services, so set that in.envfirst. -
Configure the database connection
cp .env.example .envEdit
.envand setDATABASE_URLto point to your PostgreSQL instance. -
Build and run
make build ./siloThe server starts at
http://localhost:8080by default. All other settings are configured through the admin UI.
Reporting Issues
If you are reporting a bug, install problem, or performance issue, start with the admin workflow and reproduction steps, not Claude/Codex analysis.
Please include:
- What you were trying to do
- Exact steps you took
- What you expected to happen
- What actually happened
- What exact action is slow or broken (
save,scan,browse,import,playback, etc.) - Whether it happens every time or only sometimes
- The library, media type, filter, setting, or value involved
- Version, branch, commit, and deployment details if you know them
- Screenshots, recordings, or log snippets if relevant
If you used Claude/Codex for debugging, put that under Technical notes at the end. Suspected files, SQL output, stack traces, and root-cause theories can be helpful, but only after the workflow and repro steps are clear.
Use this template:
Goal:
Steps:
Expected:
Actual:
What is slow/broken:
Scope:
Version/branch:
Deployment:
Technical notes:
Contributing & Development
Silo is open source and contributions are welcome. See DEVELOPMENT.md for building from source in a dev workflow, running tests, database migrations, and project layout, and CONTRIBUTING.md for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
Supporting Silo
Silo is an open-source hobby project, developed in spare time and funded out of pocket. If you'd like to support development, you can sponsor via GitHub Sponsors.
Donations go directly toward the costs of building and running the project:
- AI development tooling subscriptions (Claude, Codex) used to build and maintain Silo
- Push notification relay infrastructure
- Future development costs
Sponsoring is entirely optional — Silo is and will remain free and open source. Bug reports, contributions, and feedback are just as valuable.
License & Trademarks
Silo's source code is licensed under the GNU Affero General Public License
v3.0 or later (AGPL-3.0-or-later) — see LICENSE.
The Silo name, logo, and wordmark are trademarks of Silo Media L.L.C. and are not covered by the AGPL. You're free to fork and redistribute the code, but forks and redistributions must not use the Silo brand as their identity and must remove or replace the brand assets. Publishing a Silo-branded app to an app store requires written permission. See TRADEMARK.md for what's permitted — including referential use like "compatible with Silo."