SBDeveloper/medley
Watch
1
0
Fork
You've already forked medley
 0
Code Issues 16 Pull requests Releases 13 Activity Actions

YouTube video mode #93

New issue
Closed
opened 2026-10-18 18:30:16 +02:00 by SBDeveloper · 3 comments
SBDeveloper commented 2026-10-18 18:30:16 +02:00
Owner

A mode to switch between audio-only and video mode (using YouTube), if you are interested in the videoclip.

A mode to switch between audio-only and video mode (using YouTube), if you are interested in the videoclip.
SBDeveloper commented 2026-10-18 20:43:16 +02:00
Author
Owner

This was generated by AI during triage.

Triage Notes

Not implemented anywhere (searched code, docs and glossary for video/YouTube). No prior rejection. Category enhancement, state needs-triage, blocked on #94 (does the YouTube embed play in the Linux and Android webviews?).

Decisions so far (draft in docs/adr/0022-video-mode-embeds-the-youtube-player.md, term Video mode in GLOSSARY.md):

  • YouTube plugin only; no general host video concept.
  • Official YouTube IFrame embed shown in a host-owned video pane. No stream extraction, so the host audio pipeline (Crossfade, Visualizers, offline cache) is bypassed for Tracks played as video.
  • Plugin resolves Track to video ID via a small additive plugin-api lookup, using the YouTube Data API with the user's own key.
  • Silent best match, with a visible "wrong video?" correction. Match cached per Recording.
  • Video mode is sticky across the Queue and per Device. Handoff carries Queue, Track and position, not the mode. A Track with no match (or a quota error) plays as audio and the mode resumes on the next one.
  • Hidden in Offline mode. Linux and Android from the start.

Risks: YouTube may reject the embed under Tauri's webview origin (error 153); the free API quota (100 units per search) is easily hit on a fresh Library.

Once #94 is resolved, this gets an agent brief or goes to wontfix if the embed can't work.

> *This was generated by AI during triage.* ## Triage Notes Not implemented anywhere (searched code, docs and glossary for video/YouTube). No prior rejection. Category `enhancement`, state `needs-triage`, **blocked on #94** (does the YouTube embed play in the Linux and Android webviews?). **Decisions so far** (draft in `docs/adr/0022-video-mode-embeds-the-youtube-player.md`, term **Video mode** in `GLOSSARY.md`): - YouTube plugin only; no general host video concept. - Official YouTube IFrame embed shown in a host-owned video pane. No stream extraction, so the host audio pipeline (Crossfade, Visualizers, offline cache) is bypassed for Tracks played as video. - Plugin resolves Track to video ID via a small additive `plugin-api` lookup, using the YouTube Data API with the user's own key. - Silent best match, with a visible "wrong video?" correction. Match cached per Recording. - Video mode is sticky across the Queue and per Device. Handoff carries Queue, Track and position, not the mode. A Track with no match (or a quota error) plays as audio and the mode resumes on the next one. - Hidden in Offline mode. Linux and Android from the start. **Risks:** YouTube may reject the embed under Tauri's webview origin (error 153); the free API quota (100 units per search) is easily hit on a fresh Library. Once #94 is resolved, this gets an agent brief or goes to `wontfix` if the embed can't work.
SBDeveloper commented 2026-10-19 20:52:20 +02:00
Author
Owner

Agent brief

Prototype #94 settled the embed question; see ADR 0022 (accepted).

Key finding: the YouTube IFrame player gives error 153 under tauri://localhost (Linux), works under http://tauri.localhost (Android). Use a player page served from http://localhost:<port> by the shell, iframed into the video pane, driven via postMessage. Reference: branch prototype/94-youtube-embed (scripts/proto-yt/player.html, ui/src/lib/PrototypeYoutube.svelte). Do not merge it; port the idea.

Scope (follow ADR 0022 rules): host-owned video pane; additive plugin-api lookup Track -> video ID; YouTube plugin using the Data API with the user's key and network capability; match cache per Recording; "wrong video?" correction; sticky per-Device Video mode; fallback to audio on no match or quota error; hidden in Offline mode; CSP update; localhost server in src-tauri.

Acceptance: plays and advances the Queue on ended; pause/seek/position reach the host; works on Linux release build and Android release build.

## Agent brief Prototype #94 settled the embed question; see ADR 0022 (accepted). **Key finding:** the YouTube IFrame player gives error 153 under `tauri://localhost` (Linux), works under `http://tauri.localhost` (Android). Use a player page served from `http://localhost:<port>` by the shell, iframed into the video pane, driven via `postMessage`. Reference: branch `prototype/94-youtube-embed` (`scripts/proto-yt/player.html`, `ui/src/lib/PrototypeYoutube.svelte`). Do not merge it; port the idea. **Scope** (follow ADR 0022 rules): host-owned video pane; additive `plugin-api` lookup Track -> video ID; YouTube plugin using the Data API with the user's key and `network` capability; match cache per Recording; "wrong video?" correction; sticky per-Device Video mode; fallback to audio on no match or quota error; hidden in Offline mode; CSP update; localhost server in `src-tauri`. **Acceptance:** plays and advances the Queue on ended; pause/seek/position reach the host; works on Linux release build and Android release build.
SBDeveloper commented 2026-10-19 22:05:30 +02:00
Author
Owner

Implemented on branch ticket/93-youtube-video-mode (not merged yet).

What it does

  • Video mode is a per-Device setting with an Audio | Video switch in the player bar and big player; the video takes the cover's place in the big player and keeps playing out of sight when it is closed. Hidden in Offline mode.
  • Core holds each Track silent until its video is known, routes pause/resume/seek (app, media keys, MPRIS) to the video, shows the video's position as the playback state, moves the Queue on when it ends, and falls back to audio where it stopped if the video cannot play. Switching Audio <-> Video carries the position and play/pause state over.
  • New video_lookup plugin capability and plugin-api types; plugins/optional/youtube (Data API, user's own key) ranks candidates by channel, title, length and category; matches are cached per Recording; "Wrong video?" picks another candidate, "Search again" re-runs the lookup.
  • Player page served from loopback (http://localhost:<port>) because the embed refuses tauri://localhost (#94); Android release builds allow cleartext to loopback only.

Known limits

  • Visualizers, Crossfade and the offline cache do not apply to video (ADR 0022); the Visualizer tab says so.
  • Linux: WebKitGTK registers its own MPRIS player for the embed and cannot be told not to. The app's own player stays off the bus while a video is loaded, and the embed's player is fed the app's title/artist/cover and next/previous. Whether the metadata swap works was not confirmed on a running build.
  • Not yet run on release builds of Linux or Android, and the plugin's YouTube calls were not exercised against the real API in tests (Core and ranking are tested).
  • The YouTube plugin (0.1.1) is not published to the registry.

Tests: crates/core/tests/video.rs (24), ranking unit tests in the plugin. Existing failures on main unrelated to this: discord_presence, low_delay; listenbrainz is flaky.

Implemented on branch `ticket/93-youtube-video-mode` (not merged yet). **What it does** - Video mode is a per-Device setting with an Audio | Video switch in the player bar and big player; the video takes the cover's place in the big player and keeps playing out of sight when it is closed. Hidden in Offline mode. - Core holds each Track silent until its video is known, routes pause/resume/seek (app, media keys, MPRIS) to the video, shows the video's position as the playback state, moves the Queue on when it ends, and falls back to audio where it stopped if the video cannot play. Switching Audio <-> Video carries the position and play/pause state over. - New `video_lookup` plugin capability and `plugin-api` types; `plugins/optional/youtube` (Data API, user's own key) ranks candidates by channel, title, length and category; matches are cached per Recording; "Wrong video?" picks another candidate, "Search again" re-runs the lookup. - Player page served from loopback (`http://localhost:<port>`) because the embed refuses `tauri://localhost` (#94); Android release builds allow cleartext to loopback only. **Known limits** - Visualizers, Crossfade and the offline cache do not apply to video (ADR 0022); the Visualizer tab says so. - Linux: WebKitGTK registers its own MPRIS player for the embed and cannot be told not to. The app's own player stays off the bus while a video is loaded, and the embed's player is fed the app's title/artist/cover and next/previous. Whether the metadata swap works was not confirmed on a running build. - Not yet run on release builds of Linux or Android, and the plugin's YouTube calls were not exercised against the real API in tests (Core and ranking are tested). - The YouTube plugin (0.1.1) is not published to the registry. Tests: `crates/core/tests/video.rs` (24), ranking unit tests in the plugin. Existing failures on `main` unrelated to this: `discord_presence`, `low_delay`; `listenbrainz` is flaky.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
prototype/ma-live-input
v0.6.0
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.1
v0.3.2
v0.3.0
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug
Something is broken

  
enhancement
New feature or improvement

  
needs-info
Waiting on reporter for more information

  
needs-triage
Maintainer needs to evaluate this issue

  
ready-for-agent
Fully specified, ready for an AFK agent

  
ready-for-human
Requires human implementation

  
security
Security-related fix

  
wontfix
Will not be actioned

No labels
bug
enhancement
needs-info
needs-triage
ready-for-agent
ready-for-human
security
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
SBDeveloper/medley#93
No description provided.