253 lines
10 KiB
Markdown
253 lines
10 KiB
Markdown
# Karaoke Song Library Cleanup Tool — PRD (v1 CLI)
|
|
|
|
## 1. Project Summary
|
|
|
|
- **Goal:** Analyze, deduplicate, and suggest cleanup of a large karaoke song collection, outputting a JSON “skip list” (for future imports) and supporting flexible reporting and manual review.
|
|
- **Primary User:** Admin (self, collection owner)
|
|
- **Initial Interface:** Command Line (CLI) with print/logging and JSON output
|
|
- **Future Expansion:** Optional web UI for filtering, review, and playback
|
|
|
|
---
|
|
|
|
## 2. Architectural Priorities
|
|
|
|
### 2.1 Code Organization Principles
|
|
|
|
**TOP PRIORITY:** The codebase must be built with the following architectural principles from the beginning:
|
|
|
|
- **True Separation of Concerns:**
|
|
- Many small files with focused responsibilities
|
|
- Each module/class should have a single, well-defined purpose
|
|
- Avoid monolithic files with mixed responsibilities
|
|
|
|
- **Constants and Enums:**
|
|
- Create constants, enums, and configuration objects to avoid duplicate code or values
|
|
- Centralize magic numbers, strings, and configuration values
|
|
- Use enums for type safety and clarity
|
|
|
|
- **Readability and Maintainability:**
|
|
- Code should be self-documenting with clear naming conventions
|
|
- Easy to understand, extend, and refactor
|
|
- Consistent patterns throughout the codebase
|
|
|
|
- **Extensibility:**
|
|
- Design for future growth and feature additions
|
|
- Modular architecture that allows easy integration of new components
|
|
- Clear interfaces between modules
|
|
|
|
- **Refactorability:**
|
|
- Code structure should make future refactoring straightforward
|
|
- Minimize coupling between components
|
|
- Use dependency injection and abstraction where appropriate
|
|
|
|
These principles are fundamental to the project's long-term success and must be applied consistently throughout development.
|
|
|
|
---
|
|
|
|
## 3. Data Handling & Matching Logic
|
|
|
|
### 3.1 Input
|
|
|
|
- Reads from `/data/allSongs.json`
|
|
- Each song includes at least:
|
|
- `artist`, `title`, `path`, (plus id3 tag info, `channel` for MP4s)
|
|
|
|
### 3.2 Song Matching
|
|
|
|
- **Primary keys:** `artist` + `title`
|
|
- Fuzzy matching configurable (enabled/disabled with threshold)
|
|
- Multi-artist handling: parse delimiters (commas, “feat.”, etc.)
|
|
- **File type detection:** Use file extension from `path` (`.mp3`, `.cdg`, `.mp4`)
|
|
|
|
### 3.3 Channel Priority (for MP4s)
|
|
|
|
- **Configurable folder names:**
|
|
- Set in `/config/config.json` as an array of folder names
|
|
- Order = priority (first = highest priority)
|
|
- Tool searches for these folder names within the song's `path` property
|
|
- Songs without matching folder names are marked for manual review
|
|
- **File type priority:** MP4 > CDG/MP3 pairs > standalone MP3 > standalone CDG
|
|
- **CDG/MP3 pairing:** CDG and MP3 files with the same base filename are treated as a single karaoke song unit
|
|
|
|
---
|
|
|
|
## 4. Output & Reporting
|
|
|
|
### 4.1 Skip List
|
|
|
|
- **Format:** JSON (`/data/skipSongs.json`)
|
|
- List of file paths to skip in future imports
|
|
- Optionally: “reason” field (e.g., `{"path": "...", "reason": "duplicate"}`)
|
|
|
|
### 4.2 CLI Reporting
|
|
|
|
- **Summary:** Total songs, duplicates found, types breakdown, etc.
|
|
- **Verbose per-song output:** Only for matches/duplicates (not every song)
|
|
- **Verbosity configurable:** (via CLI flag or config)
|
|
|
|
### 4.3 Manual Review (Web UI)
|
|
|
|
- **Interactive Web Interface**: Table/grid view for ambiguous/complex cases
|
|
- **Media Preview**: Ability to preview media before making a selection
|
|
- **Bulk Actions**: Select multiple items for batch operations
|
|
- **Real-time Filtering**: Search and filter capabilities
|
|
- **Responsive Design**: Works on desktop and mobile devices
|
|
- **Easy Startup**: Simple script (`start_web_ui.py`) with dependency checking
|
|
|
|
---
|
|
|
|
## 5. Features & Edge Cases
|
|
|
|
- **Batch Processing:**
|
|
- E.g., "Auto-skip all but highest-priority channel for each song"
|
|
- Manual review as CLI flag (future: always in web UI)
|
|
- **Edge Cases:**
|
|
- Multiple versions (>2 formats)
|
|
- Support for keeping multiple versions per song (configurable/manual)
|
|
- **Non-destructive:** Never deletes or moves files, only generates skip list and reports
|
|
|
|
---
|
|
|
|
## 6. Tech Stack & Organization
|
|
|
|
- **CLI Language:** Python
|
|
- **Config:** JSON (channel priorities, settings)
|
|
- **Current Folder Structure:**
|
|
```
|
|
KaraokeMerge/
|
|
├── data/
|
|
│ ├── allSongs.json # Input: Your song library data
|
|
│ ├── skipSongs.json # Output: Generated skip list
|
|
│ └── reports/ # Detailed analysis reports
|
|
│ ├── analysis_data.json
|
|
│ ├── actionable_insights_report.txt
|
|
│ ├── channel_optimization_report.txt
|
|
│ ├── duplicate_pattern_report.txt
|
|
│ ├── enhanced_summary_report.txt
|
|
│ ├── skip_list_summary.txt
|
|
│ └── skip_songs_detailed.json
|
|
├── config/
|
|
│ └── config.json # Configuration settings
|
|
├── cli/
|
|
│ ├── main.py # Main CLI application
|
|
│ ├── matching.py # Song matching logic
|
|
│ ├── report.py # Report generation
|
|
│ └── utils.py # Utility functions
|
|
├── web/ # Web UI for manual review
|
|
│ ├── app.py # Flask web application
|
|
│ └── templates/
|
|
│ └── index.html # Web interface template
|
|
├── start_web_ui.py # Web UI startup script
|
|
├── test_tool.py # Validation and testing script
|
|
├── requirements.txt # Python dependencies
|
|
├── .gitignore # Git ignore rules
|
|
├── PRD.md # Product Requirements Document
|
|
└── README.md # Project documentation
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Web UI Implementation
|
|
|
|
### 7.1 Current Web UI Features
|
|
- **Interactive Table View**: Sortable, filterable grid of duplicate songs
|
|
- **Bulk Selection**: Select multiple items for batch operations
|
|
- **Search & Filter**: Real-time search across artists, titles, and paths
|
|
- **Responsive Design**: Mobile-friendly interface
|
|
- **Easy Startup**: Automated dependency checking and browser launch
|
|
|
|
### 7.2 Web UI Architecture
|
|
- **Flask Backend**: Lightweight web server (`web/app.py`)
|
|
- **HTML Template**: Modern, responsive interface (`web/templates/index.html`)
|
|
- **Startup Script**: Dependency management and server startup (`start_web_ui.py`)
|
|
|
|
### 7.3 Future Web UI Enhancements
|
|
- Embedded media player for audio/video preview
|
|
- Real-time configuration editing
|
|
- Advanced filtering and sorting options
|
|
- Export capabilities for manual selections
|
|
|
|
---
|
|
|
|
## 8. Open Questions (for future refinement)
|
|
|
|
- Fuzzy matching library/thresholds?
|
|
- Best parsing rules for multi-artist/feat. strings?
|
|
- Any alternate export formats needed?
|
|
- Temporary/partial skip support for "under review" songs?
|
|
|
|
---
|
|
|
|
## 9. Implementation Status
|
|
|
|
### ✅ Completed Features
|
|
- [x] Write initial CLI tool to parse allSongs.json, deduplicate, and output skipSongs.json
|
|
- [x] Print CLI summary reports (with verbosity control)
|
|
- [x] Implement config file support for channel priority
|
|
- [x] Organize folder/file structure for easy expansion
|
|
|
|
### 🎯 Current Implementation
|
|
The tool has been successfully implemented with the following components:
|
|
|
|
**Core Modules:**
|
|
- `cli/main.py` - Main CLI application with argument parsing
|
|
- `cli/matching.py` - Song matching and deduplication logic
|
|
- `cli/report.py` - Report generation and output formatting
|
|
- `cli/utils.py` - Utility functions for file operations and data processing
|
|
|
|
**Configuration:**
|
|
- `config/config.json` - Configurable settings for channel priorities, matching rules, and output options
|
|
|
|
**Features Implemented:**
|
|
- Multi-format support (MP3, CDG, MP4)
|
|
- **CDG/MP3 Pairing Logic**: Files with same base filename treated as single karaoke song units
|
|
- Channel priority system for MP4 files (based on folder names in path)
|
|
- Fuzzy matching support with configurable threshold
|
|
- Multi-artist parsing with various delimiters
|
|
- **Enhanced Analysis & Reporting**: Comprehensive statistical analysis with actionable insights
|
|
- Channel priority analysis and manual review identification
|
|
- Non-destructive operation (skip lists only)
|
|
- Verbose and dry-run modes
|
|
- Detailed duplicate analysis
|
|
- Skip list generation with metadata
|
|
- **Pattern Analysis**: Skip list pattern analysis and channel optimization suggestions
|
|
|
|
**File Type Priority System:**
|
|
1. **MP4 files** (with channel priority sorting)
|
|
2. **CDG/MP3 pairs** (treated as single units)
|
|
3. **Standalone MP3** files
|
|
4. **Standalone CDG** files
|
|
|
|
**Performance Results:**
|
|
- Successfully processed 37,015 songs
|
|
- Identified 12,424 duplicates (33.6% duplicate rate)
|
|
- Generated comprehensive skip list with metadata (10,998 unique files after deduplication)
|
|
- Optimized for large datasets with progress indicators
|
|
- **Enhanced Analysis**: Generated 7 detailed reports with actionable insights
|
|
- **Bug Fix**: Resolved duplicate entries in skip list (removed 1,426 duplicate entries)
|
|
|
|
### 📋 Next Steps Checklist
|
|
|
|
#### ✅ **Completed**
|
|
- [x] Write initial CLI tool to parse allSongs.json, deduplicate, and output skipSongs.json
|
|
- [x] Print CLI summary reports (with verbosity control)
|
|
- [x] Implement config file support for channel priority
|
|
- [x] Organize folder/file structure for easy expansion
|
|
- [x] Implement CDG/MP3 pairing logic for accurate duplicate detection
|
|
- [x] Generate comprehensive skip list with metadata
|
|
- [x] Optimize performance for large datasets (37,000+ songs)
|
|
- [x] Add progress indicators and error handling
|
|
- [x] Generate detailed analysis reports (`--save-reports` functionality)
|
|
- [x] Create web UI for manual review of ambiguous cases
|
|
- [x] Add test tool for validation and debugging
|
|
- [x] Create startup script for web UI with dependency checking
|
|
- [x] Add comprehensive .gitignore file
|
|
- [x] Update documentation with required data file information
|
|
|
|
#### 🎯 **Next Priority Items**
|
|
- [ ] Analyze MP4 files without channel priorities to suggest new folder names
|
|
- [ ] Add support for additional file formats if needed
|
|
- [ ] Implement batch processing capabilities
|
|
- [ ] Create integration scripts for karaoke software
|
|
- [ ] Add unit tests for core functionality
|
|
- [ ] Implement audio fingerprinting for better duplicate detection |