Playlist Update Optimization: Reduced Memory Usage

An optimization has been introduced to reduce memory consumption during playlist updates.

Overview
Previously, provider playlists were fully loaded into memory during the update process. For large playlists (e.g. hundreds of thousands of entries or multiple providers), this could result in significant RAM usage.
With the new implementation, it is now possible to optionally read provider playlists directly from disk instead of keeping them entirely in memory.

How It Works
 - users can configure whether:
     - Provider playlists are loaded into memory (previous behavior), or
     - Provider playlists are streamed to/from disk to minimize RAM usage.

Processing remains sequential and batch-based, ensuring identical functional behavior.
This approach significantly lowers peak memory usage, especially on systems with limited resources.

Benefits

- Reduced peak RAM consumption during playlist updates
- Better scalability for large playlists and multiple providers
- Full backward compatibility

Trade-offs / Drawbacks

 - Increased processing time due to reduced in-memory caching
 - Higher disk I/O usage, especially for large or fragmented playlists
 - Performance depends more strongly on disk speed (Nvme SSD, HDD)
 - Slightly increased CPU overhead due to repeated parsing and deserialization
 - Not optimal for environments where fast updates are more important than memory usage

Recommendation

 - Use in-memory mode for systems with sufficient RAM and a focus on update speed
 - Use disk-based mode for large playlists, multiple providers, or memory-constrained environments
This commit is contained in:
euzu
2026-01-08 15:54:12 +01:00
parent 8cc9283a31
commit 6f776782f1
34 changed files with 132 additions and 107 deletions
+12 -10
View File
@@ -560,17 +560,19 @@ impl PlaylistProcessingContext {
pub async fn get_input_lock(&self, input_name: &str) -> OwnedRwLockWriteGuard<()> {
let mut locks = self.input_locks.lock().await;
// Clean up stale weak references
// Try to upgrade the existing weak reference
let lock = locks.get(input_name)
.and_then(Weak::upgrade)
.unwrap_or_else(|| {
let new_lock = Arc::new(RwLock::new(()));
locks.insert(input_name.to_string(), Arc::downgrade(&new_lock));
new_lock
});
// Clean up stale references periodically
locks.retain(|_, weak| weak.strong_count() > 0);
if let Some(weak) = locks.get(input_name) {
if let Some(strong) = weak.upgrade() {
return strong.write_owned().await;
}
}
let lock = Arc::new(RwLock::new(()));
locks.insert(input_name.to_string(), Arc::downgrade(&lock));
drop(locks); // Release mutex before awaiting write lock
lock.write_owned().await
}
}
@@ -776,7 +778,7 @@ pub fn process_favourites(playlist: &mut Vec<PlaylistGroup>, favourites_cfg: Opt
let mut fav_groups: IndexMap<Arc<str>, Vec<PlaylistItem>> = IndexMap::new();
for pg in playlist.iter() {
for pli in &pg.channels {
// series episodes cant be included in favourites
// series episodes can't be included in favourites
if pli.header.item_type == PlaylistItemType::Series || pli.header.item_type == PlaylistItemType::LocalSeries {
continue;
}
+15 -3
View File
@@ -109,10 +109,22 @@ fn playlist_comparator(
}
fn playlistgroup_comparator(a: &PlaylistGroup, b: &PlaylistGroup, group_sort: &ConfigSortGroup, match_as_ascii: bool) -> Ordering {
let value_a = if match_as_ascii { deunicode(&a.title) } else { a.title.to_string() };
let value_b = if match_as_ascii { deunicode(&b.title) } else { b.title.to_string() };
let ascii_a;
let ascii_b;
let value_a: &str = if match_as_ascii {
ascii_a = deunicode(&a.title);
&ascii_a
} else {
&a.title
};
let value_b: &str = if match_as_ascii {
ascii_b = deunicode(&b.title);
&ascii_b
} else {
&b.title
};
playlist_comparator(group_sort.sequence.as_ref(), group_sort.order, &value_a, &value_b)
playlist_comparator(group_sort.sequence.as_ref(), group_sort.order, value_a, value_b)
}
fn playlistitem_comparator(