diff --git a/PRD.md b/PRD.md index aa26ada..9228000 100644 --- a/PRD.md +++ b/PRD.md @@ -1,8 +1,8 @@ -# ๐ŸŽค Karaoke Video Downloader โ€“ PRD (v3.3) +# ๐ŸŽค Karaoke Video Downloader โ€“ PRD (v3.5) ## โœ… Overview -A Python-based Windows CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp.exe`, with advanced tracking, songlist prioritization, and flexible configuration. The codebase has been comprehensively refactored into a modular architecture with centralized utilities for improved maintainability, error handling, and code reuse. +A Python-based cross-platform CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp`, with advanced tracking, songlist prioritization, and flexible configuration. Supports Windows, macOS, and Linux with automatic platform detection and optimized caching. The codebase has been comprehensively refactored into a modular architecture with centralized utilities for improved maintainability, error handling, and code reuse. --- @@ -63,9 +63,9 @@ The codebase has been refactored into focused modules with centralized utilities --- ## โš™๏ธ Platform & Stack -- **Platform:** Windows +- **Platform:** Windows, macOS, Linux - **Interface:** Command-line (CLI) -- **Tech Stack:** Python 3.7+, yt-dlp.exe, mutagen (for ID3 tagging) +- **Tech Stack:** Python 3.7+, yt-dlp (platform-specific binary), mutagen (for ID3 tagging) --- @@ -159,7 +159,9 @@ KaroakeVideoDownloader/ โ”œโ”€โ”€ downloads/ # All video output โ”‚ โ””โ”€โ”€ [ChannelName]/ # Per-channel folders โ”œโ”€โ”€ logs/ # Download logs -โ”œโ”€โ”€ downloader/yt-dlp.exe # yt-dlp binary +โ”œโ”€โ”€ downloader/yt-dlp.exe # yt-dlp binary (Windows) +โ”œโ”€โ”€ downloader/yt-dlp_macos # yt-dlp binary (macOS) +โ”œโ”€โ”€ downloader/yt-dlp # yt-dlp binary (Linux) โ”œโ”€โ”€ tests/ # Diagnostic and test scripts โ”‚ โ””โ”€โ”€ test_installation.py โ”œโ”€โ”€ download_karaoke.py # Main entry point (thin wrapper) @@ -260,6 +262,17 @@ The codebase has been comprehensively refactored to improve maintainability and - **Performance improvements:** Significantly faster downloads for large batches (3-5x speedup with 3-5 workers) - **Integrated with all modes:** Works with both songlist-across-channels and latest-per-channel download modes +### **Cross-Platform Support (v3.5)** +- **Platform detection:** Automatic detection of Windows, macOS, and Linux systems +- **Flexible yt-dlp integration:** Supports both binary files and pip-installed yt-dlp modules +- **Platform-specific configuration:** Automatic selection of appropriate yt-dlp binary/command for each platform +- **Setup automation:** `setup_platform.py` script for easy platform-specific setup +- **Command parsing:** Intelligent parsing of yt-dlp commands (file paths vs. module commands) +- **Enhanced documentation:** Platform-specific setup instructions and troubleshooting +- **Backward compatibility:** Maintains full compatibility with existing Windows installations +- **FFmpeg integration:** Automatic FFmpeg installation and configuration for optimal video processing +- **Optimized caching:** Enhanced channel video caching with format compatibility and instant video list loading + --- ## ๐Ÿš€ Future Enhancements @@ -268,6 +281,7 @@ The codebase has been comprehensively refactored to improve maintainability and - [ ] Download scheduling and retry logic - [ ] More granular status reporting - [x] **Parallel downloads for improved speed** โœ… **COMPLETED** +- [x] **Cross-platform support (Windows, macOS, Linux)** โœ… **COMPLETED** - [ ] Unit tests for all modules - [ ] Integration tests for end-to-end workflows - [ ] Plugin system for custom file operations diff --git a/README.md b/README.md index a299ed4..4c36ab8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # ๐ŸŽค Karaoke Video Downloader -A Python-based Windows CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp.exe`, with advanced tracking, songlist prioritization, and flexible configuration. +A Python-based cross-platform CLI tool to download karaoke videos from YouTube channels/playlists using `yt-dlp`, with advanced tracking, songlist prioritization, and flexible configuration. Supports Windows, macOS, and Linux with automatic platform detection, optimized caching, and FFmpeg integration. ## โœจ Features - ๐ŸŽต **Channel & Playlist Downloads**: Download all videos from a YouTube channel or playlist @@ -21,6 +21,9 @@ A Python-based Windows CLI tool to download karaoke videos from YouTube channels - โšก **Optimized Scanning**: High-performance channel scanning with O(nร—m) complexity, pre-processed lookups, and early termination for faster matching - ๐Ÿท๏ธ **Server Duplicates Tracking**: Automatically checks against local songs.json file and marks duplicates for future skipping, preventing re-downloads of songs already on the server - โšก **Parallel Downloads**: Enable concurrent downloads with `--parallel --workers N` for significantly faster batch downloads (3-5x speedup) +- ๐ŸŒ **Cross-Platform Support**: Automatic platform detection and yt-dlp integration for Windows, macOS, and Linux +- ๐Ÿš€ **Optimized Caching**: Enhanced channel video caching with instant video list loading +- ๐ŸŽฌ **FFmpeg Integration**: Automatic FFmpeg installation and configuration for optimal video processing ## ๐Ÿ—๏ธ Architecture The codebase has been comprehensively refactored into a modular architecture with centralized utilities for improved maintainability, error handling, and code reuse: @@ -80,13 +83,57 @@ The codebase has been comprehensively refactored into a modular architecture wit - **Type Safety**: Comprehensive type hints across all new modules ## ๐Ÿ“‹ Requirements -- **Windows 10/11** +- **Windows 10/11, macOS 10.14+, or Linux** - **Python 3.7+** -- **yt-dlp.exe** (in `downloader/`) +- **yt-dlp binary** (platform-specific, see setup instructions below) - **mutagen** (for ID3 tagging, optional) - **ffmpeg/ffprobe** (for video validation, optional but recommended) - **rapidfuzz** (for fuzzy matching, optional, falls back to difflib) +## ๐Ÿ–ฅ๏ธ Platform Setup + +### Automatic Setup (Recommended) +Run the platform setup script to automatically set up yt-dlp for your system: + +```bash +python setup_platform.py +``` + +This script will: +- Detect your platform (Windows, macOS, or Linux) +- Offer two installation options: + 1. **Download binary file** (recommended for most users) + 2. **Install via pip** (alternative method) +- Make binaries executable (on Unix-like systems) +- Install FFmpeg (for optimal video processing) +- Test the installation + +### Manual Setup +If you prefer to set up manually: + +#### Option 1: Download Binary Files +1. **Windows**: Download `yt-dlp.exe` from [yt-dlp releases](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.exe) +2. **macOS**: Download `yt-dlp_macos` from [yt-dlp releases](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos) +3. **Linux**: Download `yt-dlp` from [yt-dlp releases](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp) + +Place the downloaded file in the `downloader/` directory and make it executable on Unix-like systems: +```bash +chmod +x downloader/yt-dlp_macos # macOS +chmod +x downloader/yt-dlp # Linux +``` + +#### Option 2: Install via pip +```bash +pip install yt-dlp +``` + +The tool will automatically detect and use the pip-installed version on macOS. + +**Note**: FFmpeg is also required for optimal video processing. The setup script will attempt to install it automatically, or you can install it manually: +- **macOS**: `brew install ffmpeg` +- **Linux**: `sudo apt install ffmpeg` (Ubuntu/Debian) or `sudo yum install ffmpeg` (CentOS/RHEL) +- **Windows**: Download from [ffmpeg.org](https://ffmpeg.org/download.html) + ## ๐Ÿš€ Quick Start > **๐Ÿ’ก Pro Tip**: For a complete list of all available commands, see `commands.txt` - you can copy/paste any command directly into your terminal! @@ -230,7 +277,11 @@ KaroakeVideoDownloader/ โ”œโ”€โ”€ downloads/ # All video output โ”‚ โ””โ”€โ”€ [ChannelName]/ # Per-channel folders โ”œโ”€โ”€ logs/ # Download logs -โ”œโ”€โ”€ downloader/yt-dlp.exe # yt-dlp binary +โ”œโ”€โ”€ downloader/yt-dlp.exe # yt-dlp binary (Windows) +โ”œโ”€โ”€ downloader/yt-dlp_macos # yt-dlp binary (macOS) +โ”œโ”€โ”€ downloader/yt-dlp # yt-dlp binary (Linux) +โ”œโ”€โ”€ setup_platform.py # Platform setup script +โ”œโ”€โ”€ test_platform.py # Platform test script โ”œโ”€โ”€ tests/ # Diagnostic and test scripts โ”‚ โ””โ”€โ”€ test_installation.py โ”œโ”€โ”€ download_karaoke.py # Main entry point (thin wrapper) @@ -311,7 +362,7 @@ python download_karaoke.py --clear-server-duplicates > **๐Ÿ”„ Maintenance Note**: The `commands.txt` file should be kept up to date with any CLI changes. When adding new command-line options or modifying existing ones, update this file to reflect all available commands and their usage. -## ๐Ÿ”ง Refactoring Improvements (v3.3) +## ๐Ÿ”ง Refactoring Improvements (v3.5) The codebase has been comprehensively refactored to improve maintainability and reduce code duplication. Recent improvements have enhanced reliability, performance, and code organization: ### **New Utility Modules (v3.3)** @@ -346,6 +397,17 @@ The codebase has been comprehensively refactored to improve maintainability and - **Improved Testability**: Modular components can be tested independently - **Better Developer Experience**: Clear function signatures and comprehensive documentation +### **Cross-Platform Support (v3.5)** +- **Platform detection:** Automatic detection of Windows, macOS, and Linux systems +- **Flexible yt-dlp integration:** Supports both binary files and pip-installed yt-dlp modules +- **Platform-specific configuration:** Automatic selection of appropriate yt-dlp binary/command for each platform +- **Setup automation:** `setup_platform.py` script for easy platform-specific setup +- **Command parsing:** Intelligent parsing of yt-dlp commands (file paths vs. module commands) +- **Enhanced documentation:** Platform-specific setup instructions and troubleshooting +- **Backward compatibility:** Maintains full compatibility with existing Windows installations +- **FFmpeg integration:** Automatic FFmpeg installation and configuration for optimal video processing +- **Optimized caching:** Enhanced channel video caching with format compatibility and instant video list loading + ### **New Parallel Download System (v3.4)** - **Parallel downloader module:** `parallel_downloader.py` provides thread-safe concurrent download management - **Configurable concurrency:** Use `--parallel --workers N` to enable parallel downloads with N workers (1-10) @@ -372,7 +434,11 @@ The codebase has been comprehensively refactored to improve maintainability and - **Robust download plan execution:** Fixed index management in download plan execution to prevent errors during interrupted downloads. ## ๐Ÿž Troubleshooting -- Ensure `yt-dlp.exe` is in the `downloader/` folder +- **Platform-specific yt-dlp setup**: + - **Windows**: Ensure `yt-dlp.exe` is in the `downloader/` folder + - **macOS**: Either ensure `yt-dlp_macos` is in the `downloader/` folder (make executable with `chmod +x`) OR install via pip (`pip install yt-dlp`) + - **Linux**: Ensure `yt-dlp` is in the `downloader/` folder (make executable with `chmod +x`) +- Run `python setup_platform.py` to automatically set up yt-dlp for your platform - Check `logs/` for error details - Use `python -m karaoke_downloader.check_resolution` to verify video quality - If you see errors about ffmpeg/ffprobe, install [ffmpeg](https://ffmpeg.org/download.html) and ensure it is in your PATH diff --git a/commands.txt b/commands.txt index 7adfa73..b1a37fb 100644 --- a/commands.txt +++ b/commands.txt @@ -1,6 +1,6 @@ # ๐ŸŽค Karaoke Video Downloader - CLI Commands Reference # Copy and paste these commands into your terminal -# Updated: v3.4 (includes parallel downloads and all refactoring improvements) +# Updated: v3.5 (includes cross-platform support, optimized caching, and all refactoring improvements) ## ๐Ÿ“ฅ BASIC DOWNLOADS @@ -197,11 +197,27 @@ python download_karaoke.py --reset-channel SingKingKaraoke --reset-songlist python download_karaoke.py --status python download_karaoke.py --clear-cache all +## ๐ŸŒ PLATFORM SETUP COMMANDS (v3.5) + +# Automatic platform setup (detects OS and installs yt-dlp + FFmpeg) +python setup_platform.py + +# Test platform detection and yt-dlp integration +python test_platform.py + +# Manual platform-specific setup +# Windows: Download yt-dlp.exe to downloader/ folder +# macOS: brew install ffmpeg && pip install yt-dlp +# Linux: sudo apt install ffmpeg && download yt-dlp to downloader/ folder + ## ๐Ÿ”ง TROUBLESHOOTING COMMANDS # Check if everything is working python download_karaoke.py --version +# Test platform setup +python test_platform.py + # Force refresh everything python download_karaoke.py --force-download-plan --refresh --clear-cache all @@ -233,6 +249,8 @@ python download_karaoke.py --clear-server-duplicates # - Use --fuzzy-match for better song discovery # - Use --refresh sparingly (forces re-scan) # - Clear cache if you encounter issues +# - Channel caching provides instant video list loading (no YouTube API calls) +# - FFmpeg integration ensures optimal video processing and merging # Parallel download tips: # - Start with --workers 3 for conservative approach diff --git a/data/config.json b/data/config.json index 65a5ed7..973e879 100644 --- a/data/config.json +++ b/data/config.json @@ -34,5 +34,13 @@ "include_console": true, "include_file": true }, + "platform_settings": { + "auto_detect_platform": true, + "yt_dlp_paths": { + "windows": "downloader/yt-dlp.exe", + "macos": "python3 -m yt_dlp", + "linux": "downloader/yt-dlp" + } + }, "yt_dlp_path": "downloader/yt-dlp.exe" } \ No newline at end of file diff --git a/downloader/yt-dlp b/downloader/yt-dlp new file mode 100644 index 0000000..d0e858e Binary files /dev/null and b/downloader/yt-dlp differ diff --git a/downloader/yt-dlp_macos b/downloader/yt-dlp_macos new file mode 100755 index 0000000..df8a46b Binary files /dev/null and b/downloader/yt-dlp_macos differ diff --git a/karaoke_downloader/cli.py b/karaoke_downloader/cli.py index 6ecb2a7..5a47061 100644 --- a/karaoke_downloader/cli.py +++ b/karaoke_downloader/cli.py @@ -177,11 +177,37 @@ Examples: print("โŒ Error: --workers must be between 1 and 10") sys.exit(1) - yt_dlp_path = Path("downloader/yt-dlp.exe") - if not yt_dlp_path.exists(): - print("โŒ Error: yt-dlp.exe not found in downloader/ directory") - print("Please ensure yt-dlp.exe is present in the downloader/ folder") - sys.exit(1) + # Load configuration to get platform-aware yt-dlp path + from karaoke_downloader.config_manager import load_config + config = load_config() + yt_dlp_path = config.yt_dlp_path + + # Check if it's a command string (like "python3 -m yt_dlp") or a file path + if yt_dlp_path.startswith(('python', 'python3')): + # It's a command string, test if it works + try: + import subprocess + from karaoke_downloader.youtube_utils import _parse_yt_dlp_command + cmd = _parse_yt_dlp_command(yt_dlp_path) + ["--version"] + result = subprocess.run(cmd, capture_output=True, text=True, timeout=10) + if result.returncode != 0: + raise Exception(f"Command failed: {result.stderr}") + except Exception as e: + platform_name = "macOS" if sys.platform == "darwin" else "Windows" if sys.platform == "win32" else "Linux" + print(f"โŒ Error: yt-dlp command failed: {yt_dlp_path}") + print(f"Please ensure yt-dlp is properly installed for {platform_name}") + print(f"Error: {e}") + sys.exit(1) + else: + # It's a file path, check if it exists + yt_dlp_file = Path(yt_dlp_path) + if not yt_dlp_file.exists(): + platform_name = "macOS" if sys.platform == "darwin" else "Windows" if sys.platform == "win32" else "Linux" + binary_name = yt_dlp_file.name + print(f"โŒ Error: {binary_name} not found in downloader/ directory") + print(f"Please ensure {binary_name} is present in the downloader/ folder for {platform_name}") + print(f"Expected path: {yt_dlp_file}") + sys.exit(1) downloader = KaraokeDownloader() diff --git a/karaoke_downloader/config_manager.py b/karaoke_downloader/config_manager.py index 3007b41..3e5b5cd 100644 --- a/karaoke_downloader/config_manager.py +++ b/karaoke_downloader/config_manager.py @@ -4,6 +4,8 @@ Provides centralized configuration loading, validation, and management. """ import json +import platform +import sys from dataclasses import dataclass, field from datetime import datetime from pathlib import Path @@ -42,6 +44,14 @@ DEFAULT_CONFIG = { "include_console": True, "include_file": True, }, + "platform_settings": { + "auto_detect_platform": True, + "yt_dlp_paths": { + "windows": "downloader/yt-dlp.exe", + "macos": "downloader/yt-dlp_macos", + "linux": "downloader/yt-dlp", + }, + }, "yt_dlp_path": "downloader/yt-dlp.exe", } @@ -55,6 +65,26 @@ RESOLUTION_MAP = { } +def detect_platform() -> str: + """Detect the current platform and return the appropriate platform key.""" + system = platform.system().lower() + if system == "windows": + return "windows" + elif system == "darwin": + return "macos" + elif system == "linux": + return "linux" + else: + # Default to windows for unknown platforms + return "windows" + + +def get_platform_yt_dlp_path(platform_paths: Dict[str, str]) -> str: + """Get the appropriate yt-dlp path for the current platform.""" + platform_key = detect_platform() + return platform_paths.get(platform_key, platform_paths.get("windows", "downloader/yt-dlp.exe")) + + @dataclass class DownloadSettings: """Configuration for download settings.""" @@ -234,11 +264,21 @@ class ConfigManager: folder_structure = FolderStructure(**config_data.get("folder_structure", {})) logging_config = LoggingConfig(**config_data.get("logging", {})) + # Handle platform-specific yt-dlp path + yt_dlp_path = config_data.get("yt_dlp_path", "downloader/yt-dlp.exe") + + # Check if platform auto-detection is enabled + platform_settings = config_data.get("platform_settings", {}) + if platform_settings.get("auto_detect_platform", True): + platform_paths = platform_settings.get("yt_dlp_paths", {}) + if platform_paths: + yt_dlp_path = get_platform_yt_dlp_path(platform_paths) + return AppConfig( download_settings=download_settings, folder_structure=folder_structure, logging=logging_config, - yt_dlp_path=config_data.get("yt_dlp_path", "downloader/yt-dlp.exe"), + yt_dlp_path=yt_dlp_path, _config_file=self.config_file, ) diff --git a/karaoke_downloader/downloader.py b/karaoke_downloader/downloader.py index 5bc3cb1..528c8dd 100644 --- a/karaoke_downloader/downloader.py +++ b/karaoke_downloader/downloader.py @@ -78,7 +78,7 @@ class KaraokeDownloader: self.config = self.config_manager.load_config() # Initialize paths - self.yt_dlp_path = Path(self.config.yt_dlp_path) + self.yt_dlp_path = self.config.yt_dlp_path # Keep as string for command parsing self.downloads_dir = Path(self.config.folder_structure.downloads_dir) self.logs_dir = Path(self.config.folder_structure.logs_dir) @@ -191,25 +191,11 @@ class KaraokeDownloader: server_duplicates_tracking = load_server_duplicates_tracking() limit = getattr(self.config, "limit", 1) - cmd = [ - str(self.yt_dlp_path), - "--flat-playlist", - "--print", - "%(title)s|%(id)s|%(url)s", - url, - ] - try: - result = subprocess.run(cmd, capture_output=True, text=True, check=True) - lines = result.stdout.strip().splitlines() - except subprocess.CalledProcessError as e: - print(f"โŒ yt-dlp failed to fetch playlist: {e}") - return False - available_videos = [] - for line in lines: - parts = line.split("|") - if len(parts) >= 2: - title, video_id = parts[0].strip(), parts[1].strip() - available_videos.append({"title": title, "id": video_id}) + + # Use tracking manager's cache instead of calling yt-dlp directly + available_videos = self.tracker.get_channel_video_list( + url, yt_dlp_path=self.yt_dlp_path, force_refresh=force_refresh + ) # Normalize songlist for matching normalized_songlist = { create_song_key(s["artist"], s["title"]): s for s in songlist @@ -282,7 +268,7 @@ class KaraokeDownloader: return True # Download only the first N matches using the new pipeline pipeline = DownloadPipeline( - yt_dlp_path=str(self.yt_dlp_path), + yt_dlp_path=self.yt_dlp_path, config=self.config, downloads_dir=self.downloads_dir, songlist_tracking=self.songlist_tracking, @@ -538,7 +524,7 @@ class KaraokeDownloader: # Create parallel downloader parallel_downloader = create_parallel_downloader( - yt_dlp_path=str(self.yt_dlp_path), + yt_dlp_path=self.yt_dlp_path, config=self.config, downloads_dir=self.downloads_dir, max_workers=self.parallel_workers, @@ -620,7 +606,7 @@ class KaraokeDownloader: # Create parallel downloader parallel_downloader = create_parallel_downloader( - yt_dlp_path=str(self.yt_dlp_path), + yt_dlp_path=self.yt_dlp_path, config=self.config, downloads_dir=self.downloads_dir, max_workers=self.parallel_workers, @@ -769,7 +755,7 @@ class KaraokeDownloader: # Use the new pipeline for consistent processing pipeline = DownloadPipeline( - yt_dlp_path=str(self.yt_dlp_path), + yt_dlp_path=self.yt_dlp_path, config=self.config, downloads_dir=self.downloads_dir, songlist_tracking=self.songlist_tracking, @@ -874,7 +860,7 @@ class KaraokeDownloader: channel_name, channel_id = get_channel_info(channel_url) print(f"\n๐Ÿšฆ Starting channel: {channel_name} ({channel_url})") available_videos = self.tracker.get_channel_video_list( - channel_url, yt_dlp_path=str(self.yt_dlp_path), force_refresh=False + channel_url, yt_dlp_path=self.yt_dlp_path, force_refresh=False ) print( f" โ†’ Found {len(available_videos)} total videos for this channel." diff --git a/karaoke_downloader/resolution_cli.py b/karaoke_downloader/resolution_cli.py index e1a8fa3..7986aa9 100644 --- a/karaoke_downloader/resolution_cli.py +++ b/karaoke_downloader/resolution_cli.py @@ -56,6 +56,14 @@ def update_resolution(resolution): "include_console": True, "include_file": True, }, + "platform_settings": { + "auto_detect_platform": True, + "yt_dlp_paths": { + "windows": "downloader/yt-dlp.exe", + "macos": "downloader/yt-dlp_macos", + "linux": "downloader/yt-dlp", + }, + }, "yt_dlp_path": "downloader/yt-dlp.exe", } diff --git a/karaoke_downloader/tracking_manager.py b/karaoke_downloader/tracking_manager.py index 9644359..3e20130 100644 --- a/karaoke_downloader/tracking_manager.py +++ b/karaoke_downloader/tracking_manager.py @@ -283,28 +283,54 @@ class TrackingManager: self._save() def get_channel_video_list( - self, channel_url, yt_dlp_path="downloader/yt-dlp.exe", force_refresh=False + self, channel_url, yt_dlp_path=None, force_refresh=False ): """ Return a list of videos (dicts with 'title' and 'id') for the channel, using cache if available unless force_refresh is True. """ + # Use platform-aware path if none provided + if yt_dlp_path is None: + from karaoke_downloader.config_manager import load_config + config = load_config() + yt_dlp_path = config.yt_dlp_path + channel_name, channel_id = None, None from karaoke_downloader.youtube_utils import get_channel_info channel_name, channel_id = get_channel_info(channel_url) - # Try multiple possible cache keys - possible_keys = [ - channel_id, # The extracted channel ID - channel_url, # The full URL - channel_name, # The extracted channel name - ] - + # Check if cache has the old flat structure or new nested structure + cache_data = None cache_key = None - for key in possible_keys: - if key and key in self.cache: - cache_key = key - break + + # Try nested structure first (new format) + if "channels" in self.cache: + # Try multiple possible cache keys in nested structure + possible_keys = [ + channel_id, # The extracted channel ID + channel_url, # The full URL + channel_name, # The extracted channel name + ] + + for key in possible_keys: + if key and key in self.cache["channels"]: + cache_data = self.cache["channels"][key]["videos"] + cache_key = key + break + + # Try flat structure (old format) as fallback + if cache_data is None: + possible_keys = [ + channel_id, # The extracted channel ID + channel_url, # The full URL + channel_name, # The extracted channel name + ] + + for key in possible_keys: + if key and key in self.cache: + cache_data = self.cache[key] + cache_key = key + break if not cache_key: cache_key = channel_id or channel_url # Use as fallback for new entries @@ -312,19 +338,31 @@ class TrackingManager: print(f" ๐Ÿ” Trying cache keys: {possible_keys}") print(f" ๐Ÿ” Selected cache key: '{cache_key}'") - if not force_refresh and cache_key in self.cache: + if not force_refresh and cache_data is not None: print( - f" ๐Ÿ“‹ Using cached video list ({len(self.cache[cache_key])} videos)" + f" ๐Ÿ“‹ Using cached video list ({len(cache_data)} videos)" ) - return self.cache[cache_key] + # Convert old cache format to new format if needed + converted_videos = [] + for video in cache_data: + if "video_id" in video and "id" not in video: + # Convert old format to new format + converted_videos.append({ + "title": video["title"], + "id": video["video_id"] + }) + else: + # Already in new format + converted_videos.append(video) + return converted_videos else: print(f" โŒ Cache miss for all keys") # Fetch with yt-dlp print(f" ๐ŸŒ Fetching video list from YouTube (this may take a while)...") import subprocess + from karaoke_downloader.youtube_utils import _parse_yt_dlp_command - cmd = [ - yt_dlp_path, + cmd = _parse_yt_dlp_command(yt_dlp_path) + [ "--flat-playlist", "--print", "%(title)s|%(id)s|%(url)s", @@ -339,7 +377,18 @@ class TrackingManager: if len(parts) >= 2: title, video_id = parts[0].strip(), parts[1].strip() videos.append({"title": title, "id": video_id}) - self.cache[cache_key] = videos + + # Save in nested structure format + if "channels" not in self.cache: + self.cache["channels"] = {} + + self.cache["channels"][cache_key] = { + "videos": videos, + "last_updated": datetime.now().isoformat(), + "channel_name": channel_name, + "channel_id": channel_id + } + self.save_cache() return videos except subprocess.CalledProcessError as e: diff --git a/karaoke_downloader/youtube_utils.py b/karaoke_downloader/youtube_utils.py index bb65cab..571b875 100644 --- a/karaoke_downloader/youtube_utils.py +++ b/karaoke_downloader/youtube_utils.py @@ -9,10 +9,29 @@ from typing import Any, Dict, List, Optional, Union from karaoke_downloader.config_manager import AppConfig +def _parse_yt_dlp_command(yt_dlp_path: str) -> List[str]: + """ + Parse yt-dlp path/command into a list of command arguments. + Handles both file paths and command strings like 'python3 -m yt_dlp'. + """ + if yt_dlp_path.startswith(('python', 'python3')): + # It's a Python module command + return yt_dlp_path.split() + else: + # It's a file path + return [yt_dlp_path] + + def get_channel_info( - channel_url: str, yt_dlp_path: str = "downloader/yt-dlp.exe" + channel_url: str, yt_dlp_path: str = None ) -> tuple[str, str]: """Get channel information using yt-dlp. Returns (channel_name, channel_id).""" + # Use platform-aware path if none provided + if yt_dlp_path is None: + from karaoke_downloader.config_manager import load_config + config = load_config() + yt_dlp_path = config.yt_dlp_path + try: # Extract channel name from URL for now (faster than calling yt-dlp) if "/@" in channel_url: @@ -39,11 +58,17 @@ def get_channel_info( def get_playlist_info( - playlist_url: str, yt_dlp_path: str = "downloader/yt-dlp.exe" + playlist_url: str, yt_dlp_path: str = None ) -> List[Dict[str, Any]]: """Get playlist information using yt-dlp.""" + # Use platform-aware path if none provided + if yt_dlp_path is None: + from karaoke_downloader.config_manager import load_config + config = load_config() + yt_dlp_path = config.yt_dlp_path + try: - cmd = [yt_dlp_path, "--dump-json", "--flat-playlist", playlist_url] + cmd = _parse_yt_dlp_command(yt_dlp_path) + ["--dump-json", "--flat-playlist", playlist_url] result = subprocess.run(cmd, capture_output=True, text=True, check=True) videos = [] for line in result.stdout.strip().split("\n"): @@ -75,8 +100,7 @@ def build_yt_dlp_command( Returns: List of command arguments for subprocess.run """ - cmd = [ - str(yt_dlp_path), + cmd = _parse_yt_dlp_command(str(yt_dlp_path)) + [ "--no-check-certificates", "--ignore-errors", "--no-warnings", @@ -117,7 +141,7 @@ def execute_yt_dlp_command( def show_available_formats( - video_url: str, yt_dlp_path: str = "downloader/yt-dlp.exe", timeout: int = 30 + video_url: str, yt_dlp_path: str = None, timeout: int = 30 ) -> None: """ Show available formats for a video (debugging utility). @@ -127,8 +151,14 @@ def show_available_formats( yt_dlp_path: Path to yt-dlp executable timeout: Timeout in seconds """ + # Use platform-aware path if none provided + if yt_dlp_path is None: + from karaoke_downloader.config_manager import load_config + config = load_config() + yt_dlp_path = config.yt_dlp_path + print(f"๐Ÿ” Checking available formats for: {video_url}") - format_cmd = [str(yt_dlp_path), "--list-formats", video_url] + format_cmd = _parse_yt_dlp_command(str(yt_dlp_path)) + ["--list-formats", video_url] try: format_result = subprocess.run( format_cmd, capture_output=True, text=True, timeout=timeout diff --git a/setup_platform.py b/setup_platform.py new file mode 100644 index 0000000..f799f39 --- /dev/null +++ b/setup_platform.py @@ -0,0 +1,288 @@ +#!/usr/bin/env python3 +""" +Platform setup script for Karaoke Video Downloader. +This script helps users download the correct yt-dlp binary for their platform. +""" + +import os +import platform +import sys +import urllib.request +import zipfile +import tarfile +from pathlib import Path + + +def detect_platform(): + """Detect the current platform and return platform info.""" + system = platform.system().lower() + machine = platform.machine().lower() + + if system == "windows": + return "windows", "yt-dlp.exe" + elif system == "darwin": + return "macos", "yt-dlp_macos" + elif system == "linux": + return "linux", "yt-dlp" + else: + return "unknown", "yt-dlp" + + +def get_download_url(platform_name): + """Get the download URL for yt-dlp based on platform.""" + base_url = "https://github.com/yt-dlp/yt-dlp/releases/latest/download" + + if platform_name == "windows": + return f"{base_url}/yt-dlp.exe" + elif platform_name == "macos": + return f"{base_url}/yt-dlp_macos" + elif platform_name == "linux": + return f"{base_url}/yt-dlp" + else: + raise ValueError(f"Unsupported platform: {platform_name}") + + +def install_via_pip(): + """Install yt-dlp via pip.""" + print("๐Ÿ“ฆ Installing yt-dlp via pip...") + try: + import subprocess + result = subprocess.run([sys.executable, "-m", "pip", "install", "yt-dlp"], + capture_output=True, text=True, check=True) + print("โœ… yt-dlp installed successfully via pip!") + return True + except subprocess.CalledProcessError as e: + print(f"โŒ Failed to install yt-dlp via pip: {e}") + return False + + +def check_ffmpeg(): + """Check if FFmpeg is installed and available.""" + try: + import subprocess + result = subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True, timeout=10) + return result.returncode == 0 + except (subprocess.TimeoutExpired, FileNotFoundError): + return False + + +def install_ffmpeg(): + """Install FFmpeg based on platform.""" + import subprocess + platform_name, _ = detect_platform() + + print("๐ŸŽฌ Installing FFmpeg...") + + if platform_name == "macos": + # Try using Homebrew first + try: + print("๐Ÿบ Attempting to install FFmpeg via Homebrew...") + result = subprocess.run(["brew", "install", "ffmpeg"], + capture_output=True, text=True, check=True) + print("โœ… FFmpeg installed successfully via Homebrew!") + return True + except (subprocess.CalledProcessError, FileNotFoundError): + print("โš ๏ธ Homebrew not found or failed. Trying alternative methods...") + + # Try using MacPorts + try: + print("๐ŸŽ Attempting to install FFmpeg via MacPorts...") + result = subprocess.run(["sudo", "port", "install", "ffmpeg"], + capture_output=True, text=True, check=True) + print("โœ… FFmpeg installed successfully via MacPorts!") + return True + except (subprocess.CalledProcessError, FileNotFoundError): + print("โŒ Could not install FFmpeg automatically.") + print("Please install FFmpeg manually:") + print("1. Install Homebrew: https://brew.sh/") + print("2. Run: brew install ffmpeg") + print("3. Or download from: https://ffmpeg.org/download.html") + return False + + elif platform_name == "linux": + try: + print("๐Ÿง Attempting to install FFmpeg via package manager...") + # Try apt (Ubuntu/Debian) + try: + result = subprocess.run(["sudo", "apt", "update"], capture_output=True, text=True, check=True) + result = subprocess.run(["sudo", "apt", "install", "-y", "ffmpeg"], + capture_output=True, text=True, check=True) + print("โœ… FFmpeg installed successfully via apt!") + return True + except subprocess.CalledProcessError: + # Try yum (CentOS/RHEL) + try: + result = subprocess.run(["sudo", "yum", "install", "-y", "ffmpeg"], + capture_output=True, text=True, check=True) + print("โœ… FFmpeg installed successfully via yum!") + return True + except subprocess.CalledProcessError: + print("โŒ Could not install FFmpeg automatically.") + print("Please install FFmpeg manually for your Linux distribution.") + return False + except FileNotFoundError: + print("โŒ Could not install FFmpeg automatically.") + print("Please install FFmpeg manually for your Linux distribution.") + return False + + elif platform_name == "windows": + print("โŒ FFmpeg installation not automated for Windows.") + print("Please install FFmpeg manually:") + print("1. Download from: https://ffmpeg.org/download.html") + print("2. Extract to a folder and add to PATH") + print("3. Or use Chocolatey: choco install ffmpeg") + return False + + return False + + +def download_file(url, destination): + """Download a file from URL to destination.""" + print(f"๐Ÿ“ฅ Downloading from: {url}") + print(f"๐Ÿ“ Saving to: {destination}") + + try: + urllib.request.urlretrieve(url, destination) + print("โœ… Download completed successfully!") + return True + except Exception as e: + print(f"โŒ Download failed: {e}") + return False + + +def make_executable(file_path): + """Make a file executable (for Unix-like systems).""" + try: + os.chmod(file_path, 0o755) + print(f"๐Ÿ”ง Made {file_path} executable") + except Exception as e: + print(f"โš ๏ธ Could not make file executable: {e}") + + +def main(): + print("๐ŸŽค Karaoke Video Downloader - Platform Setup") + print("=" * 50) + + # Detect platform + platform_name, binary_name = detect_platform() + print(f"๐Ÿ–ฅ๏ธ Detected platform: {platform_name}") + print(f"๐Ÿ“ฆ Binary name: {binary_name}") + + # Create downloader directory if it doesn't exist + downloader_dir = Path("downloader") + downloader_dir.mkdir(exist_ok=True) + + # Check if binary already exists + binary_path = downloader_dir / binary_name + if binary_path.exists(): + print(f"โœ… {binary_name} already exists in downloader/ directory") + response = input("Do you want to re-download it? (y/N): ").strip().lower() + if response != 'y': + print("Setup completed!") + return + + # Offer installation options + print(f"\n๐Ÿ”ง Installation options for {platform_name}:") + print("1. Download binary file (recommended for most users)") + print("2. Install via pip (alternative method)") + + choice = input("Choose installation method (1 or 2): ").strip() + + if choice == "2": + # Install via pip + if install_via_pip(): + print(f"\nโœ… yt-dlp installed successfully!") + + # Test the installation + print(f"\n๐Ÿงช Testing yt-dlp installation...") + try: + import subprocess + result = subprocess.run([sys.executable, "-m", "yt_dlp", "--version"], + capture_output=True, text=True, timeout=10) + if result.returncode == 0: + version = result.stdout.strip() + print(f"โœ… yt-dlp is working! Version: {version}") + else: + print(f"โš ๏ธ yt-dlp test failed: {result.stderr}") + except Exception as e: + print(f"โš ๏ธ Could not test yt-dlp: {e}") + + # Check and install FFmpeg + print(f"\n๐ŸŽฌ Checking FFmpeg installation...") + if check_ffmpeg(): + print(f"โœ… FFmpeg is already installed and working!") + else: + print(f"โš ๏ธ FFmpeg not found. Installing...") + if install_ffmpeg(): + print(f"โœ… FFmpeg installed successfully!") + else: + print(f"โš ๏ธ FFmpeg installation failed. The tool will still work but may be slower.") + + print(f"\n๐ŸŽ‰ Setup completed successfully!") + print(f"๐Ÿ“ฆ yt-dlp installed via pip") + print(f"๐Ÿ–ฅ๏ธ Platform: {platform_name}") + print(f"\n๐ŸŽ‰ You're ready to use the Karaoke Video Downloader!") + print(f"Run: python download_karaoke.py --help") + return + else: + print("โŒ Pip installation failed. Trying binary download...") + + # Download binary file + try: + download_url = get_download_url(platform_name) + except ValueError as e: + print(f"โŒ {e}") + print("Please manually download yt-dlp for your platform from:") + print("https://github.com/yt-dlp/yt-dlp/releases/latest") + return + + # Download the binary + print(f"\n๐Ÿš€ Downloading yt-dlp for {platform_name}...") + if download_file(download_url, binary_path): + # Make executable on Unix-like systems + if platform_name in ["macos", "linux"]: + make_executable(binary_path) + + print(f"\nโœ… yt-dlp binary downloaded successfully!") + print(f"๐Ÿ“ yt-dlp binary location: {binary_path}") + print(f"๐Ÿ–ฅ๏ธ Platform: {platform_name}") + + # Test the binary + print(f"\n๐Ÿงช Testing yt-dlp installation...") + try: + import subprocess + result = subprocess.run([str(binary_path), "--version"], + capture_output=True, text=True, timeout=10) + if result.returncode == 0: + version = result.stdout.strip() + print(f"โœ… yt-dlp is working! Version: {version}") + else: + print(f"โš ๏ธ yt-dlp test failed: {result.stderr}") + except Exception as e: + print(f"โš ๏ธ Could not test yt-dlp: {e}") + + # Check and install FFmpeg + print(f"\n๐ŸŽฌ Checking FFmpeg installation...") + if check_ffmpeg(): + print(f"โœ… FFmpeg is already installed and working!") + else: + print(f"โš ๏ธ FFmpeg not found. Installing...") + if install_ffmpeg(): + print(f"โœ… FFmpeg installed successfully!") + else: + print(f"โš ๏ธ FFmpeg installation failed. The tool will still work but may be slower.") + + print(f"\n๐ŸŽ‰ Setup completed successfully!") + print(f"๐Ÿ“ yt-dlp binary location: {binary_path}") + print(f"๐Ÿ–ฅ๏ธ Platform: {platform_name}") + print(f"\n๐ŸŽ‰ You're ready to use the Karaoke Video Downloader!") + print(f"Run: python download_karaoke.py --help") + + else: + print(f"\nโŒ Setup failed. Please manually download yt-dlp for {platform_name}") + print(f"Download URL: {download_url}") + print(f"Save to: {binary_path}") + + +if __name__ == "__main__": + main() \ No newline at end of file