Compare commits

...

1 Commits

Author SHA1 Message Date
bed46ff2d2 Signed-off-by: Matt Bruce <mbrucedogs@gmail.com> 2025-08-05 12:59:09 -05:00
13 changed files with 601 additions and 68 deletions

24
PRD.md
View File

@ -1,8 +1,8 @@
# 🎤 Karaoke Video Downloader PRD (v3.3) # 🎤 Karaoke Video Downloader PRD (v3.5)
## ✅ Overview ## ✅ Overview
A Python-based Windows CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp.exe`, with advanced tracking, songlist prioritization, and flexible configuration. The codebase has been comprehensively refactored into a modular architecture with centralized utilities for improved maintainability, error handling, and code reuse. A Python-based cross-platform CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp`, with advanced tracking, songlist prioritization, and flexible configuration. Supports Windows, macOS, and Linux with automatic platform detection and optimized caching. The codebase has been comprehensively refactored into a modular architecture with centralized utilities for improved maintainability, error handling, and code reuse.
--- ---
@ -63,9 +63,9 @@ The codebase has been refactored into focused modules with centralized utilities
--- ---
## ⚙️ Platform & Stack ## ⚙️ Platform & Stack
- **Platform:** Windows - **Platform:** Windows, macOS, Linux
- **Interface:** Command-line (CLI) - **Interface:** Command-line (CLI)
- **Tech Stack:** Python 3.7+, yt-dlp.exe, mutagen (for ID3 tagging) - **Tech Stack:** Python 3.7+, yt-dlp (platform-specific binary), mutagen (for ID3 tagging)
--- ---
@ -159,7 +159,9 @@ KaroakeVideoDownloader/
├── downloads/ # All video output ├── downloads/ # All video output
│ └── [ChannelName]/ # Per-channel folders │ └── [ChannelName]/ # Per-channel folders
├── logs/ # Download logs ├── logs/ # Download logs
├── downloader/yt-dlp.exe # yt-dlp binary ├── downloader/yt-dlp.exe # yt-dlp binary (Windows)
├── downloader/yt-dlp_macos # yt-dlp binary (macOS)
├── downloader/yt-dlp # yt-dlp binary (Linux)
├── tests/ # Diagnostic and test scripts ├── tests/ # Diagnostic and test scripts
│ └── test_installation.py │ └── test_installation.py
├── download_karaoke.py # Main entry point (thin wrapper) ├── download_karaoke.py # Main entry point (thin wrapper)
@ -260,6 +262,17 @@ The codebase has been comprehensively refactored to improve maintainability and
- **Performance improvements:** Significantly faster downloads for large batches (3-5x speedup with 3-5 workers) - **Performance improvements:** Significantly faster downloads for large batches (3-5x speedup with 3-5 workers)
- **Integrated with all modes:** Works with both songlist-across-channels and latest-per-channel download modes - **Integrated with all modes:** Works with both songlist-across-channels and latest-per-channel download modes
### **Cross-Platform Support (v3.5)**
- **Platform detection:** Automatic detection of Windows, macOS, and Linux systems
- **Flexible yt-dlp integration:** Supports both binary files and pip-installed yt-dlp modules
- **Platform-specific configuration:** Automatic selection of appropriate yt-dlp binary/command for each platform
- **Setup automation:** `setup_platform.py` script for easy platform-specific setup
- **Command parsing:** Intelligent parsing of yt-dlp commands (file paths vs. module commands)
- **Enhanced documentation:** Platform-specific setup instructions and troubleshooting
- **Backward compatibility:** Maintains full compatibility with existing Windows installations
- **FFmpeg integration:** Automatic FFmpeg installation and configuration for optimal video processing
- **Optimized caching:** Enhanced channel video caching with format compatibility and instant video list loading
--- ---
## 🚀 Future Enhancements ## 🚀 Future Enhancements
@ -268,6 +281,7 @@ The codebase has been comprehensively refactored to improve maintainability and
- [ ] Download scheduling and retry logic - [ ] Download scheduling and retry logic
- [ ] More granular status reporting - [ ] More granular status reporting
- [x] **Parallel downloads for improved speed** ✅ **COMPLETED** - [x] **Parallel downloads for improved speed** ✅ **COMPLETED**
- [x] **Cross-platform support (Windows, macOS, Linux)** ✅ **COMPLETED**
- [ ] Unit tests for all modules - [ ] Unit tests for all modules
- [ ] Integration tests for end-to-end workflows - [ ] Integration tests for end-to-end workflows
- [ ] Plugin system for custom file operations - [ ] Plugin system for custom file operations

View File

@ -1,6 +1,6 @@
# 🎤 Karaoke Video Downloader # 🎤 Karaoke Video Downloader
A Python-based Windows CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp.exe`, with advanced tracking, songlist prioritization, and flexible configuration. A Python-based cross-platform CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp`, with advanced tracking, songlist prioritization, and flexible configuration. Supports Windows, macOS, and Linux with automatic platform detection, optimized caching, and FFmpeg integration.
## ✨ Features ## ✨ Features
- 🎵 **Channel & Playlist Downloads**: Download all videos from a YouTube channel or playlist - 🎵 **Channel & Playlist Downloads**: Download all videos from a YouTube channel or playlist
@ -21,6 +21,9 @@ A Python-based Windows CLI tool to download karaoke videos from YouTube channels
- ⚡ **Optimized Scanning**: High-performance channel scanning with O(n×m) complexity, pre-processed lookups, and early termination for faster matching - ⚡ **Optimized Scanning**: High-performance channel scanning with O(n×m) complexity, pre-processed lookups, and early termination for faster matching
- 🏷️ **Server Duplicates Tracking**: Automatically checks against local songs.json file and marks duplicates for future skipping, preventing re-downloads of songs already on the server - 🏷️ **Server Duplicates Tracking**: Automatically checks against local songs.json file and marks duplicates for future skipping, preventing re-downloads of songs already on the server
- ⚡ **Parallel Downloads**: Enable concurrent downloads with `--parallel --workers N` for significantly faster batch downloads (3-5x speedup) - ⚡ **Parallel Downloads**: Enable concurrent downloads with `--parallel --workers N` for significantly faster batch downloads (3-5x speedup)
- 🌐 **Cross-Platform Support**: Automatic platform detection and yt-dlp integration for Windows, macOS, and Linux
- 🚀 **Optimized Caching**: Enhanced channel video caching with instant video list loading
- 🎬 **FFmpeg Integration**: Automatic FFmpeg installation and configuration for optimal video processing
## 🏗️ Architecture ## 🏗️ Architecture
The codebase has been comprehensively refactored into a modular architecture with centralized utilities for improved maintainability, error handling, and code reuse: The codebase has been comprehensively refactored into a modular architecture with centralized utilities for improved maintainability, error handling, and code reuse:
@ -80,13 +83,57 @@ The codebase has been comprehensively refactored into a modular architecture wit
- **Type Safety**: Comprehensive type hints across all new modules - **Type Safety**: Comprehensive type hints across all new modules
## 📋 Requirements ## 📋 Requirements
- **Windows 10/11** - **Windows 10/11, macOS 10.14+, or Linux**
- **Python 3.7+** - **Python 3.7+**
- **yt-dlp.exe** (in `downloader/`) - **yt-dlp binary** (platform-specific, see setup instructions below)
- **mutagen** (for ID3 tagging, optional) - **mutagen** (for ID3 tagging, optional)
- **ffmpeg/ffprobe** (for video validation, optional but recommended) - **ffmpeg/ffprobe** (for video validation, optional but recommended)
- **rapidfuzz** (for fuzzy matching, optional, falls back to difflib) - **rapidfuzz** (for fuzzy matching, optional, falls back to difflib)
## 🖥️ Platform Setup
### Automatic Setup (Recommended)
Run the platform setup script to automatically set up yt-dlp for your system:
```bash
python setup_platform.py
```
This script will:
- Detect your platform (Windows, macOS, or Linux)
- Offer two installation options:
1. **Download binary file** (recommended for most users)
2. **Install via pip** (alternative method)
- Make binaries executable (on Unix-like systems)
- Install FFmpeg (for optimal video processing)
- Test the installation
### Manual Setup
If you prefer to set up manually:
#### Option 1: Download Binary Files
1. **Windows**: Download `yt-dlp.exe` from [yt-dlp releases](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.exe)
2. **macOS**: Download `yt-dlp_macos` from [yt-dlp releases](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos)
3. **Linux**: Download `yt-dlp` from [yt-dlp releases](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp)
Place the downloaded file in the `downloader/` directory and make it executable on Unix-like systems:
```bash
chmod +x downloader/yt-dlp_macos # macOS
chmod +x downloader/yt-dlp # Linux
```
#### Option 2: Install via pip
```bash
pip install yt-dlp
```
The tool will automatically detect and use the pip-installed version on macOS.
**Note**: FFmpeg is also required for optimal video processing. The setup script will attempt to install it automatically, or you can install it manually:
- **macOS**: `brew install ffmpeg`
- **Linux**: `sudo apt install ffmpeg` (Ubuntu/Debian) or `sudo yum install ffmpeg` (CentOS/RHEL)
- **Windows**: Download from [ffmpeg.org](https://ffmpeg.org/download.html)
## 🚀 Quick Start ## 🚀 Quick Start
> **💡 Pro Tip**: For a complete list of all available commands, see `commands.txt` - you can copy/paste any command directly into your terminal! > **💡 Pro Tip**: For a complete list of all available commands, see `commands.txt` - you can copy/paste any command directly into your terminal!
@ -230,7 +277,11 @@ KaroakeVideoDownloader/
├── downloads/ # All video output ├── downloads/ # All video output
│ └── [ChannelName]/ # Per-channel folders │ └── [ChannelName]/ # Per-channel folders
├── logs/ # Download logs ├── logs/ # Download logs
├── downloader/yt-dlp.exe # yt-dlp binary ├── downloader/yt-dlp.exe # yt-dlp binary (Windows)
├── downloader/yt-dlp_macos # yt-dlp binary (macOS)
├── downloader/yt-dlp # yt-dlp binary (Linux)
├── setup_platform.py # Platform setup script
├── test_platform.py # Platform test script
├── tests/ # Diagnostic and test scripts ├── tests/ # Diagnostic and test scripts
│ └── test_installation.py │ └── test_installation.py
├── download_karaoke.py # Main entry point (thin wrapper) ├── download_karaoke.py # Main entry point (thin wrapper)
@ -311,7 +362,7 @@ python download_karaoke.py --clear-server-duplicates
> **🔄 Maintenance Note**: The `commands.txt` file should be kept up to date with any CLI changes. When adding new command-line options or modifying existing ones, update this file to reflect all available commands and their usage. > **🔄 Maintenance Note**: The `commands.txt` file should be kept up to date with any CLI changes. When adding new command-line options or modifying existing ones, update this file to reflect all available commands and their usage.
## 🔧 Refactoring Improvements (v3.3) ## 🔧 Refactoring Improvements (v3.5)
The codebase has been comprehensively refactored to improve maintainability and reduce code duplication. Recent improvements have enhanced reliability, performance, and code organization: The codebase has been comprehensively refactored to improve maintainability and reduce code duplication. Recent improvements have enhanced reliability, performance, and code organization:
### **New Utility Modules (v3.3)** ### **New Utility Modules (v3.3)**
@ -346,6 +397,17 @@ The codebase has been comprehensively refactored to improve maintainability and
- **Improved Testability**: Modular components can be tested independently - **Improved Testability**: Modular components can be tested independently
- **Better Developer Experience**: Clear function signatures and comprehensive documentation - **Better Developer Experience**: Clear function signatures and comprehensive documentation
### **Cross-Platform Support (v3.5)**
- **Platform detection:** Automatic detection of Windows, macOS, and Linux systems
- **Flexible yt-dlp integration:** Supports both binary files and pip-installed yt-dlp modules
- **Platform-specific configuration:** Automatic selection of appropriate yt-dlp binary/command for each platform
- **Setup automation:** `setup_platform.py` script for easy platform-specific setup
- **Command parsing:** Intelligent parsing of yt-dlp commands (file paths vs. module commands)
- **Enhanced documentation:** Platform-specific setup instructions and troubleshooting
- **Backward compatibility:** Maintains full compatibility with existing Windows installations
- **FFmpeg integration:** Automatic FFmpeg installation and configuration for optimal video processing
- **Optimized caching:** Enhanced channel video caching with format compatibility and instant video list loading
### **New Parallel Download System (v3.4)** ### **New Parallel Download System (v3.4)**
- **Parallel downloader module:** `parallel_downloader.py` provides thread-safe concurrent download management - **Parallel downloader module:** `parallel_downloader.py` provides thread-safe concurrent download management
- **Configurable concurrency:** Use `--parallel --workers N` to enable parallel downloads with N workers (1-10) - **Configurable concurrency:** Use `--parallel --workers N` to enable parallel downloads with N workers (1-10)
@ -372,7 +434,11 @@ The codebase has been comprehensively refactored to improve maintainability and
- **Robust download plan execution:** Fixed index management in download plan execution to prevent errors during interrupted downloads. - **Robust download plan execution:** Fixed index management in download plan execution to prevent errors during interrupted downloads.
## 🐞 Troubleshooting ## 🐞 Troubleshooting
- Ensure `yt-dlp.exe` is in the `downloader/` folder - **Platform-specific yt-dlp setup**:
- **Windows**: Ensure `yt-dlp.exe` is in the `downloader/` folder
- **macOS**: Either ensure `yt-dlp_macos` is in the `downloader/` folder (make executable with `chmod +x`) OR install via pip (`pip install yt-dlp`)
- **Linux**: Ensure `yt-dlp` is in the `downloader/` folder (make executable with `chmod +x`)
- Run `python setup_platform.py` to automatically set up yt-dlp for your platform
- Check `logs/` for error details - Check `logs/` for error details
- Use `python -m karaoke_downloader.check_resolution` to verify video quality - Use `python -m karaoke_downloader.check_resolution` to verify video quality
- If you see errors about ffmpeg/ffprobe, install [ffmpeg](https://ffmpeg.org/download.html) and ensure it is in your PATH - If you see errors about ffmpeg/ffprobe, install [ffmpeg](https://ffmpeg.org/download.html) and ensure it is in your PATH

View File

@ -1,6 +1,6 @@
# 🎤 Karaoke Video Downloader - CLI Commands Reference # 🎤 Karaoke Video Downloader - CLI Commands Reference
# Copy and paste these commands into your terminal # Copy and paste these commands into your terminal
# Updated: v3.4 (includes parallel downloads and all refactoring improvements) # Updated: v3.5 (includes cross-platform support, optimized caching, and all refactoring improvements)
## 📥 BASIC DOWNLOADS ## 📥 BASIC DOWNLOADS
@ -197,11 +197,27 @@ python download_karaoke.py --reset-channel SingKingKaraoke --reset-songlist
python download_karaoke.py --status python download_karaoke.py --status
python download_karaoke.py --clear-cache all python download_karaoke.py --clear-cache all
## 🌐 PLATFORM SETUP COMMANDS (v3.5)
# Automatic platform setup (detects OS and installs yt-dlp + FFmpeg)
python setup_platform.py
# Test platform detection and yt-dlp integration
python test_platform.py
# Manual platform-specific setup
# Windows: Download yt-dlp.exe to downloader/ folder
# macOS: brew install ffmpeg && pip install yt-dlp
# Linux: sudo apt install ffmpeg && download yt-dlp to downloader/ folder
## 🔧 TROUBLESHOOTING COMMANDS ## 🔧 TROUBLESHOOTING COMMANDS
# Check if everything is working # Check if everything is working
python download_karaoke.py --version python download_karaoke.py --version
# Test platform setup
python test_platform.py
# Force refresh everything # Force refresh everything
python download_karaoke.py --force-download-plan --refresh --clear-cache all python download_karaoke.py --force-download-plan --refresh --clear-cache all
@ -233,6 +249,8 @@ python download_karaoke.py --clear-server-duplicates
# - Use --fuzzy-match for better song discovery # - Use --fuzzy-match for better song discovery
# - Use --refresh sparingly (forces re-scan) # - Use --refresh sparingly (forces re-scan)
# - Clear cache if you encounter issues # - Clear cache if you encounter issues
# - Channel caching provides instant video list loading (no YouTube API calls)
# - FFmpeg integration ensures optimal video processing and merging
# Parallel download tips: # Parallel download tips:
# - Start with --workers 3 for conservative approach # - Start with --workers 3 for conservative approach

View File

@ -34,5 +34,13 @@
"include_console": true, "include_console": true,
"include_file": true "include_file": true
}, },
"platform_settings": {
"auto_detect_platform": true,
"yt_dlp_paths": {
"windows": "downloader/yt-dlp.exe",
"macos": "python3 -m yt_dlp",
"linux": "downloader/yt-dlp"
}
},
"yt_dlp_path": "downloader/yt-dlp.exe" "yt_dlp_path": "downloader/yt-dlp.exe"
} }

BIN
downloader/yt-dlp Normal file

Binary file not shown.

BIN
downloader/yt-dlp_macos Executable file

Binary file not shown.

View File

@ -177,11 +177,37 @@ Examples:
print("❌ Error: --workers must be between 1 and 10") print("❌ Error: --workers must be between 1 and 10")
sys.exit(1) sys.exit(1)
yt_dlp_path = Path("downloader/yt-dlp.exe") # Load configuration to get platform-aware yt-dlp path
if not yt_dlp_path.exists(): from karaoke_downloader.config_manager import load_config
print("❌ Error: yt-dlp.exe not found in downloader/ directory") config = load_config()
print("Please ensure yt-dlp.exe is present in the downloader/ folder") yt_dlp_path = config.yt_dlp_path
sys.exit(1)
# Check if it's a command string (like "python3 -m yt_dlp") or a file path
if yt_dlp_path.startswith(('python', 'python3')):
# It's a command string, test if it works
try:
import subprocess
from karaoke_downloader.youtube_utils import _parse_yt_dlp_command
cmd = _parse_yt_dlp_command(yt_dlp_path) + ["--version"]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
if result.returncode != 0:
raise Exception(f"Command failed: {result.stderr}")
except Exception as e:
platform_name = "macOS" if sys.platform == "darwin" else "Windows" if sys.platform == "win32" else "Linux"
print(f"❌ Error: yt-dlp command failed: {yt_dlp_path}")
print(f"Please ensure yt-dlp is properly installed for {platform_name}")
print(f"Error: {e}")
sys.exit(1)
else:
# It's a file path, check if it exists
yt_dlp_file = Path(yt_dlp_path)
if not yt_dlp_file.exists():
platform_name = "macOS" if sys.platform == "darwin" else "Windows" if sys.platform == "win32" else "Linux"
binary_name = yt_dlp_file.name
print(f"❌ Error: {binary_name} not found in downloader/ directory")
print(f"Please ensure {binary_name} is present in the downloader/ folder for {platform_name}")
print(f"Expected path: {yt_dlp_file}")
sys.exit(1)
downloader = KaraokeDownloader() downloader = KaraokeDownloader()

View File

@ -4,6 +4,8 @@ Provides centralized configuration loading, validation, and management.
""" """
import json import json
import platform
import sys
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
@ -42,6 +44,14 @@ DEFAULT_CONFIG = {
"include_console": True, "include_console": True,
"include_file": True, "include_file": True,
}, },
"platform_settings": {
"auto_detect_platform": True,
"yt_dlp_paths": {
"windows": "downloader/yt-dlp.exe",
"macos": "downloader/yt-dlp_macos",
"linux": "downloader/yt-dlp",
},
},
"yt_dlp_path": "downloader/yt-dlp.exe", "yt_dlp_path": "downloader/yt-dlp.exe",
} }
@ -55,6 +65,26 @@ RESOLUTION_MAP = {
} }
def detect_platform() -> str:
"""Detect the current platform and return the appropriate platform key."""
system = platform.system().lower()
if system == "windows":
return "windows"
elif system == "darwin":
return "macos"
elif system == "linux":
return "linux"
else:
# Default to windows for unknown platforms
return "windows"
def get_platform_yt_dlp_path(platform_paths: Dict[str, str]) -> str:
"""Get the appropriate yt-dlp path for the current platform."""
platform_key = detect_platform()
return platform_paths.get(platform_key, platform_paths.get("windows", "downloader/yt-dlp.exe"))
@dataclass @dataclass
class DownloadSettings: class DownloadSettings:
"""Configuration for download settings.""" """Configuration for download settings."""
@ -234,11 +264,21 @@ class ConfigManager:
folder_structure = FolderStructure(**config_data.get("folder_structure", {})) folder_structure = FolderStructure(**config_data.get("folder_structure", {}))
logging_config = LoggingConfig(**config_data.get("logging", {})) logging_config = LoggingConfig(**config_data.get("logging", {}))
# Handle platform-specific yt-dlp path
yt_dlp_path = config_data.get("yt_dlp_path", "downloader/yt-dlp.exe")
# Check if platform auto-detection is enabled
platform_settings = config_data.get("platform_settings", {})
if platform_settings.get("auto_detect_platform", True):
platform_paths = platform_settings.get("yt_dlp_paths", {})
if platform_paths:
yt_dlp_path = get_platform_yt_dlp_path(platform_paths)
return AppConfig( return AppConfig(
download_settings=download_settings, download_settings=download_settings,
folder_structure=folder_structure, folder_structure=folder_structure,
logging=logging_config, logging=logging_config,
yt_dlp_path=config_data.get("yt_dlp_path", "downloader/yt-dlp.exe"), yt_dlp_path=yt_dlp_path,
_config_file=self.config_file, _config_file=self.config_file,
) )

View File

@ -78,7 +78,7 @@ class KaraokeDownloader:
self.config = self.config_manager.load_config() self.config = self.config_manager.load_config()
# Initialize paths # Initialize paths
self.yt_dlp_path = Path(self.config.yt_dlp_path) self.yt_dlp_path = self.config.yt_dlp_path # Keep as string for command parsing
self.downloads_dir = Path(self.config.folder_structure.downloads_dir) self.downloads_dir = Path(self.config.folder_structure.downloads_dir)
self.logs_dir = Path(self.config.folder_structure.logs_dir) self.logs_dir = Path(self.config.folder_structure.logs_dir)
@ -191,25 +191,11 @@ class KaraokeDownloader:
server_duplicates_tracking = load_server_duplicates_tracking() server_duplicates_tracking = load_server_duplicates_tracking()
limit = getattr(self.config, "limit", 1) limit = getattr(self.config, "limit", 1)
cmd = [
str(self.yt_dlp_path), # Use tracking manager's cache instead of calling yt-dlp directly
"--flat-playlist", available_videos = self.tracker.get_channel_video_list(
"--print", url, yt_dlp_path=self.yt_dlp_path, force_refresh=force_refresh
"%(title)s|%(id)s|%(url)s", )
url,
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
lines = result.stdout.strip().splitlines()
except subprocess.CalledProcessError as e:
print(f"❌ yt-dlp failed to fetch playlist: {e}")
return False
available_videos = []
for line in lines:
parts = line.split("|")
if len(parts) >= 2:
title, video_id = parts[0].strip(), parts[1].strip()
available_videos.append({"title": title, "id": video_id})
# Normalize songlist for matching # Normalize songlist for matching
normalized_songlist = { normalized_songlist = {
create_song_key(s["artist"], s["title"]): s for s in songlist create_song_key(s["artist"], s["title"]): s for s in songlist
@ -282,7 +268,7 @@ class KaraokeDownloader:
return True return True
# Download only the first N matches using the new pipeline # Download only the first N matches using the new pipeline
pipeline = DownloadPipeline( pipeline = DownloadPipeline(
yt_dlp_path=str(self.yt_dlp_path), yt_dlp_path=self.yt_dlp_path,
config=self.config, config=self.config,
downloads_dir=self.downloads_dir, downloads_dir=self.downloads_dir,
songlist_tracking=self.songlist_tracking, songlist_tracking=self.songlist_tracking,
@ -538,7 +524,7 @@ class KaraokeDownloader:
# Create parallel downloader # Create parallel downloader
parallel_downloader = create_parallel_downloader( parallel_downloader = create_parallel_downloader(
yt_dlp_path=str(self.yt_dlp_path), yt_dlp_path=self.yt_dlp_path,
config=self.config, config=self.config,
downloads_dir=self.downloads_dir, downloads_dir=self.downloads_dir,
max_workers=self.parallel_workers, max_workers=self.parallel_workers,
@ -620,7 +606,7 @@ class KaraokeDownloader:
# Create parallel downloader # Create parallel downloader
parallel_downloader = create_parallel_downloader( parallel_downloader = create_parallel_downloader(
yt_dlp_path=str(self.yt_dlp_path), yt_dlp_path=self.yt_dlp_path,
config=self.config, config=self.config,
downloads_dir=self.downloads_dir, downloads_dir=self.downloads_dir,
max_workers=self.parallel_workers, max_workers=self.parallel_workers,
@ -769,7 +755,7 @@ class KaraokeDownloader:
# Use the new pipeline for consistent processing # Use the new pipeline for consistent processing
pipeline = DownloadPipeline( pipeline = DownloadPipeline(
yt_dlp_path=str(self.yt_dlp_path), yt_dlp_path=self.yt_dlp_path,
config=self.config, config=self.config,
downloads_dir=self.downloads_dir, downloads_dir=self.downloads_dir,
songlist_tracking=self.songlist_tracking, songlist_tracking=self.songlist_tracking,
@ -874,7 +860,7 @@ class KaraokeDownloader:
channel_name, channel_id = get_channel_info(channel_url) channel_name, channel_id = get_channel_info(channel_url)
print(f"\n🚦 Starting channel: {channel_name} ({channel_url})") print(f"\n🚦 Starting channel: {channel_name} ({channel_url})")
available_videos = self.tracker.get_channel_video_list( available_videos = self.tracker.get_channel_video_list(
channel_url, yt_dlp_path=str(self.yt_dlp_path), force_refresh=False channel_url, yt_dlp_path=self.yt_dlp_path, force_refresh=False
) )
print( print(
f" → Found {len(available_videos)} total videos for this channel." f" → Found {len(available_videos)} total videos for this channel."

View File

@ -56,6 +56,14 @@ def update_resolution(resolution):
"include_console": True, "include_console": True,
"include_file": True, "include_file": True,
}, },
"platform_settings": {
"auto_detect_platform": True,
"yt_dlp_paths": {
"windows": "downloader/yt-dlp.exe",
"macos": "downloader/yt-dlp_macos",
"linux": "downloader/yt-dlp",
},
},
"yt_dlp_path": "downloader/yt-dlp.exe", "yt_dlp_path": "downloader/yt-dlp.exe",
} }

View File

@ -283,28 +283,54 @@ class TrackingManager:
self._save() self._save()
def get_channel_video_list( def get_channel_video_list(
self, channel_url, yt_dlp_path="downloader/yt-dlp.exe", force_refresh=False self, channel_url, yt_dlp_path=None, force_refresh=False
): ):
""" """
Return a list of videos (dicts with 'title' and 'id') for the channel, using cache if available unless force_refresh is True. Return a list of videos (dicts with 'title' and 'id') for the channel, using cache if available unless force_refresh is True.
""" """
# Use platform-aware path if none provided
if yt_dlp_path is None:
from karaoke_downloader.config_manager import load_config
config = load_config()
yt_dlp_path = config.yt_dlp_path
channel_name, channel_id = None, None channel_name, channel_id = None, None
from karaoke_downloader.youtube_utils import get_channel_info from karaoke_downloader.youtube_utils import get_channel_info
channel_name, channel_id = get_channel_info(channel_url) channel_name, channel_id = get_channel_info(channel_url)
# Try multiple possible cache keys # Check if cache has the old flat structure or new nested structure
possible_keys = [ cache_data = None
channel_id, # The extracted channel ID
channel_url, # The full URL
channel_name, # The extracted channel name
]
cache_key = None cache_key = None
for key in possible_keys:
if key and key in self.cache: # Try nested structure first (new format)
cache_key = key if "channels" in self.cache:
break # Try multiple possible cache keys in nested structure
possible_keys = [
channel_id, # The extracted channel ID
channel_url, # The full URL
channel_name, # The extracted channel name
]
for key in possible_keys:
if key and key in self.cache["channels"]:
cache_data = self.cache["channels"][key]["videos"]
cache_key = key
break
# Try flat structure (old format) as fallback
if cache_data is None:
possible_keys = [
channel_id, # The extracted channel ID
channel_url, # The full URL
channel_name, # The extracted channel name
]
for key in possible_keys:
if key and key in self.cache:
cache_data = self.cache[key]
cache_key = key
break
if not cache_key: if not cache_key:
cache_key = channel_id or channel_url # Use as fallback for new entries cache_key = channel_id or channel_url # Use as fallback for new entries
@ -312,19 +338,31 @@ class TrackingManager:
print(f" 🔍 Trying cache keys: {possible_keys}") print(f" 🔍 Trying cache keys: {possible_keys}")
print(f" 🔍 Selected cache key: '{cache_key}'") print(f" 🔍 Selected cache key: '{cache_key}'")
if not force_refresh and cache_key in self.cache: if not force_refresh and cache_data is not None:
print( print(
f" 📋 Using cached video list ({len(self.cache[cache_key])} videos)" f" 📋 Using cached video list ({len(cache_data)} videos)"
) )
return self.cache[cache_key] # Convert old cache format to new format if needed
converted_videos = []
for video in cache_data:
if "video_id" in video and "id" not in video:
# Convert old format to new format
converted_videos.append({
"title": video["title"],
"id": video["video_id"]
})
else:
# Already in new format
converted_videos.append(video)
return converted_videos
else: else:
print(f" ❌ Cache miss for all keys") print(f" ❌ Cache miss for all keys")
# Fetch with yt-dlp # Fetch with yt-dlp
print(f" 🌐 Fetching video list from YouTube (this may take a while)...") print(f" 🌐 Fetching video list from YouTube (this may take a while)...")
import subprocess import subprocess
from karaoke_downloader.youtube_utils import _parse_yt_dlp_command
cmd = [ cmd = _parse_yt_dlp_command(yt_dlp_path) + [
yt_dlp_path,
"--flat-playlist", "--flat-playlist",
"--print", "--print",
"%(title)s|%(id)s|%(url)s", "%(title)s|%(id)s|%(url)s",
@ -339,7 +377,18 @@ class TrackingManager:
if len(parts) >= 2: if len(parts) >= 2:
title, video_id = parts[0].strip(), parts[1].strip() title, video_id = parts[0].strip(), parts[1].strip()
videos.append({"title": title, "id": video_id}) videos.append({"title": title, "id": video_id})
self.cache[cache_key] = videos
# Save in nested structure format
if "channels" not in self.cache:
self.cache["channels"] = {}
self.cache["channels"][cache_key] = {
"videos": videos,
"last_updated": datetime.now().isoformat(),
"channel_name": channel_name,
"channel_id": channel_id
}
self.save_cache() self.save_cache()
return videos return videos
except subprocess.CalledProcessError as e: except subprocess.CalledProcessError as e:

View File

@ -9,10 +9,29 @@ from typing import Any, Dict, List, Optional, Union
from karaoke_downloader.config_manager import AppConfig from karaoke_downloader.config_manager import AppConfig
def _parse_yt_dlp_command(yt_dlp_path: str) -> List[str]:
"""
Parse yt-dlp path/command into a list of command arguments.
Handles both file paths and command strings like 'python3 -m yt_dlp'.
"""
if yt_dlp_path.startswith(('python', 'python3')):
# It's a Python module command
return yt_dlp_path.split()
else:
# It's a file path
return [yt_dlp_path]
def get_channel_info( def get_channel_info(
channel_url: str, yt_dlp_path: str = "downloader/yt-dlp.exe" channel_url: str, yt_dlp_path: str = None
) -> tuple[str, str]: ) -> tuple[str, str]:
"""Get channel information using yt-dlp. Returns (channel_name, channel_id).""" """Get channel information using yt-dlp. Returns (channel_name, channel_id)."""
# Use platform-aware path if none provided
if yt_dlp_path is None:
from karaoke_downloader.config_manager import load_config
config = load_config()
yt_dlp_path = config.yt_dlp_path
try: try:
# Extract channel name from URL for now (faster than calling yt-dlp) # Extract channel name from URL for now (faster than calling yt-dlp)
if "/@" in channel_url: if "/@" in channel_url:
@ -39,11 +58,17 @@ def get_channel_info(
def get_playlist_info( def get_playlist_info(
playlist_url: str, yt_dlp_path: str = "downloader/yt-dlp.exe" playlist_url: str, yt_dlp_path: str = None
) -> List[Dict[str, Any]]: ) -> List[Dict[str, Any]]:
"""Get playlist information using yt-dlp.""" """Get playlist information using yt-dlp."""
# Use platform-aware path if none provided
if yt_dlp_path is None:
from karaoke_downloader.config_manager import load_config
config = load_config()
yt_dlp_path = config.yt_dlp_path
try: try:
cmd = [yt_dlp_path, "--dump-json", "--flat-playlist", playlist_url] cmd = _parse_yt_dlp_command(yt_dlp_path) + ["--dump-json", "--flat-playlist", playlist_url]
result = subprocess.run(cmd, capture_output=True, text=True, check=True) result = subprocess.run(cmd, capture_output=True, text=True, check=True)
videos = [] videos = []
for line in result.stdout.strip().split("\n"): for line in result.stdout.strip().split("\n"):
@ -75,8 +100,7 @@ def build_yt_dlp_command(
Returns: Returns:
List of command arguments for subprocess.run List of command arguments for subprocess.run
""" """
cmd = [ cmd = _parse_yt_dlp_command(str(yt_dlp_path)) + [
str(yt_dlp_path),
"--no-check-certificates", "--no-check-certificates",
"--ignore-errors", "--ignore-errors",
"--no-warnings", "--no-warnings",
@ -117,7 +141,7 @@ def execute_yt_dlp_command(
def show_available_formats( def show_available_formats(
video_url: str, yt_dlp_path: str = "downloader/yt-dlp.exe", timeout: int = 30 video_url: str, yt_dlp_path: str = None, timeout: int = 30
) -> None: ) -> None:
""" """
Show available formats for a video (debugging utility). Show available formats for a video (debugging utility).
@ -127,8 +151,14 @@ def show_available_formats(
yt_dlp_path: Path to yt-dlp executable yt_dlp_path: Path to yt-dlp executable
timeout: Timeout in seconds timeout: Timeout in seconds
""" """
# Use platform-aware path if none provided
if yt_dlp_path is None:
from karaoke_downloader.config_manager import load_config
config = load_config()
yt_dlp_path = config.yt_dlp_path
print(f"🔍 Checking available formats for: {video_url}") print(f"🔍 Checking available formats for: {video_url}")
format_cmd = [str(yt_dlp_path), "--list-formats", video_url] format_cmd = _parse_yt_dlp_command(str(yt_dlp_path)) + ["--list-formats", video_url]
try: try:
format_result = subprocess.run( format_result = subprocess.run(
format_cmd, capture_output=True, text=True, timeout=timeout format_cmd, capture_output=True, text=True, timeout=timeout

288
setup_platform.py Normal file
View File

@ -0,0 +1,288 @@
#!/usr/bin/env python3
"""
Platform setup script for Karaoke Video Downloader.
This script helps users download the correct yt-dlp binary for their platform.
"""
import os
import platform
import sys
import urllib.request
import zipfile
import tarfile
from pathlib import Path
def detect_platform():
"""Detect the current platform and return platform info."""
system = platform.system().lower()
machine = platform.machine().lower()
if system == "windows":
return "windows", "yt-dlp.exe"
elif system == "darwin":
return "macos", "yt-dlp_macos"
elif system == "linux":
return "linux", "yt-dlp"
else:
return "unknown", "yt-dlp"
def get_download_url(platform_name):
"""Get the download URL for yt-dlp based on platform."""
base_url = "https://github.com/yt-dlp/yt-dlp/releases/latest/download"
if platform_name == "windows":
return f"{base_url}/yt-dlp.exe"
elif platform_name == "macos":
return f"{base_url}/yt-dlp_macos"
elif platform_name == "linux":
return f"{base_url}/yt-dlp"
else:
raise ValueError(f"Unsupported platform: {platform_name}")
def install_via_pip():
"""Install yt-dlp via pip."""
print("📦 Installing yt-dlp via pip...")
try:
import subprocess
result = subprocess.run([sys.executable, "-m", "pip", "install", "yt-dlp"],
capture_output=True, text=True, check=True)
print("✅ yt-dlp installed successfully via pip!")
return True
except subprocess.CalledProcessError as e:
print(f"❌ Failed to install yt-dlp via pip: {e}")
return False
def check_ffmpeg():
"""Check if FFmpeg is installed and available."""
try:
import subprocess
result = subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True, timeout=10)
return result.returncode == 0
except (subprocess.TimeoutExpired, FileNotFoundError):
return False
def install_ffmpeg():
"""Install FFmpeg based on platform."""
import subprocess
platform_name, _ = detect_platform()
print("🎬 Installing FFmpeg...")
if platform_name == "macos":
# Try using Homebrew first
try:
print("🍺 Attempting to install FFmpeg via Homebrew...")
result = subprocess.run(["brew", "install", "ffmpeg"],
capture_output=True, text=True, check=True)
print("✅ FFmpeg installed successfully via Homebrew!")
return True
except (subprocess.CalledProcessError, FileNotFoundError):
print("⚠️ Homebrew not found or failed. Trying alternative methods...")
# Try using MacPorts
try:
print("🍎 Attempting to install FFmpeg via MacPorts...")
result = subprocess.run(["sudo", "port", "install", "ffmpeg"],
capture_output=True, text=True, check=True)
print("✅ FFmpeg installed successfully via MacPorts!")
return True
except (subprocess.CalledProcessError, FileNotFoundError):
print("❌ Could not install FFmpeg automatically.")
print("Please install FFmpeg manually:")
print("1. Install Homebrew: https://brew.sh/")
print("2. Run: brew install ffmpeg")
print("3. Or download from: https://ffmpeg.org/download.html")
return False
elif platform_name == "linux":
try:
print("🐧 Attempting to install FFmpeg via package manager...")
# Try apt (Ubuntu/Debian)
try:
result = subprocess.run(["sudo", "apt", "update"], capture_output=True, text=True, check=True)
result = subprocess.run(["sudo", "apt", "install", "-y", "ffmpeg"],
capture_output=True, text=True, check=True)
print("✅ FFmpeg installed successfully via apt!")
return True
except subprocess.CalledProcessError:
# Try yum (CentOS/RHEL)
try:
result = subprocess.run(["sudo", "yum", "install", "-y", "ffmpeg"],
capture_output=True, text=True, check=True)
print("✅ FFmpeg installed successfully via yum!")
return True
except subprocess.CalledProcessError:
print("❌ Could not install FFmpeg automatically.")
print("Please install FFmpeg manually for your Linux distribution.")
return False
except FileNotFoundError:
print("❌ Could not install FFmpeg automatically.")
print("Please install FFmpeg manually for your Linux distribution.")
return False
elif platform_name == "windows":
print("❌ FFmpeg installation not automated for Windows.")
print("Please install FFmpeg manually:")
print("1. Download from: https://ffmpeg.org/download.html")
print("2. Extract to a folder and add to PATH")
print("3. Or use Chocolatey: choco install ffmpeg")
return False
return False
def download_file(url, destination):
"""Download a file from URL to destination."""
print(f"📥 Downloading from: {url}")
print(f"📁 Saving to: {destination}")
try:
urllib.request.urlretrieve(url, destination)
print("✅ Download completed successfully!")
return True
except Exception as e:
print(f"❌ Download failed: {e}")
return False
def make_executable(file_path):
"""Make a file executable (for Unix-like systems)."""
try:
os.chmod(file_path, 0o755)
print(f"🔧 Made {file_path} executable")
except Exception as e:
print(f"⚠️ Could not make file executable: {e}")
def main():
print("🎤 Karaoke Video Downloader - Platform Setup")
print("=" * 50)
# Detect platform
platform_name, binary_name = detect_platform()
print(f"🖥️ Detected platform: {platform_name}")
print(f"📦 Binary name: {binary_name}")
# Create downloader directory if it doesn't exist
downloader_dir = Path("downloader")
downloader_dir.mkdir(exist_ok=True)
# Check if binary already exists
binary_path = downloader_dir / binary_name
if binary_path.exists():
print(f"{binary_name} already exists in downloader/ directory")
response = input("Do you want to re-download it? (y/N): ").strip().lower()
if response != 'y':
print("Setup completed!")
return
# Offer installation options
print(f"\n🔧 Installation options for {platform_name}:")
print("1. Download binary file (recommended for most users)")
print("2. Install via pip (alternative method)")
choice = input("Choose installation method (1 or 2): ").strip()
if choice == "2":
# Install via pip
if install_via_pip():
print(f"\n✅ yt-dlp installed successfully!")
# Test the installation
print(f"\n🧪 Testing yt-dlp installation...")
try:
import subprocess
result = subprocess.run([sys.executable, "-m", "yt_dlp", "--version"],
capture_output=True, text=True, timeout=10)
if result.returncode == 0:
version = result.stdout.strip()
print(f"✅ yt-dlp is working! Version: {version}")
else:
print(f"⚠️ yt-dlp test failed: {result.stderr}")
except Exception as e:
print(f"⚠️ Could not test yt-dlp: {e}")
# Check and install FFmpeg
print(f"\n🎬 Checking FFmpeg installation...")
if check_ffmpeg():
print(f"✅ FFmpeg is already installed and working!")
else:
print(f"⚠️ FFmpeg not found. Installing...")
if install_ffmpeg():
print(f"✅ FFmpeg installed successfully!")
else:
print(f"⚠️ FFmpeg installation failed. The tool will still work but may be slower.")
print(f"\n🎉 Setup completed successfully!")
print(f"📦 yt-dlp installed via pip")
print(f"🖥️ Platform: {platform_name}")
print(f"\n🎉 You're ready to use the Karaoke Video Downloader!")
print(f"Run: python download_karaoke.py --help")
return
else:
print("❌ Pip installation failed. Trying binary download...")
# Download binary file
try:
download_url = get_download_url(platform_name)
except ValueError as e:
print(f"{e}")
print("Please manually download yt-dlp for your platform from:")
print("https://github.com/yt-dlp/yt-dlp/releases/latest")
return
# Download the binary
print(f"\n🚀 Downloading yt-dlp for {platform_name}...")
if download_file(download_url, binary_path):
# Make executable on Unix-like systems
if platform_name in ["macos", "linux"]:
make_executable(binary_path)
print(f"\n✅ yt-dlp binary downloaded successfully!")
print(f"📁 yt-dlp binary location: {binary_path}")
print(f"🖥️ Platform: {platform_name}")
# Test the binary
print(f"\n🧪 Testing yt-dlp installation...")
try:
import subprocess
result = subprocess.run([str(binary_path), "--version"],
capture_output=True, text=True, timeout=10)
if result.returncode == 0:
version = result.stdout.strip()
print(f"✅ yt-dlp is working! Version: {version}")
else:
print(f"⚠️ yt-dlp test failed: {result.stderr}")
except Exception as e:
print(f"⚠️ Could not test yt-dlp: {e}")
# Check and install FFmpeg
print(f"\n🎬 Checking FFmpeg installation...")
if check_ffmpeg():
print(f"✅ FFmpeg is already installed and working!")
else:
print(f"⚠️ FFmpeg not found. Installing...")
if install_ffmpeg():
print(f"✅ FFmpeg installed successfully!")
else:
print(f"⚠️ FFmpeg installation failed. The tool will still work but may be slower.")
print(f"\n🎉 Setup completed successfully!")
print(f"📁 yt-dlp binary location: {binary_path}")
print(f"🖥️ Platform: {platform_name}")
print(f"\n🎉 You're ready to use the Karaoke Video Downloader!")
print(f"Run: python download_karaoke.py --help")
else:
print(f"\n❌ Setup failed. Please manually download yt-dlp for {platform_name}")
print(f"Download URL: {download_url}")
print(f"Save to: {binary_path}")
if __name__ == "__main__":
main()