diff --git a/README.md b/README.md index 300b467..2a8063e 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,10 @@ MusicCharts/ - **Year Selection**: Choose different years to view available chart dates - **Date Selection**: Browse and select specific chart dates for detailed song data - **Music Chart Display**: View ranked songs with title and artist information -- **CSV Download**: Export chart data as CSV files +- **Yearly Analytics**: Calculate and display top songs of the year using multi-factor ranking algorithm +- **Enhanced Data Export**: Export chart data in JSON format with customizable properties +- **JSON Viewer**: Interactive JSON data inspection with copy/download functionality +- **Property Selection**: Choose which data properties to include in exports - **Responsive Design**: Mobile-friendly interface - **Error Handling**: Comprehensive error states and loading indicators - **Web Scraping**: Real-time data scraping from Music Charts Archive @@ -42,7 +45,10 @@ MusicCharts/ - `YearSelector`: Dropdown for year selection - `DateList`: Grid of available chart dates for selected year - `ChartTable`: Music chart data table with rankings -- `DownloadButton`: CSV export functionality +- `YearlyTopSongs`: Enhanced yearly analytics display with calculation metrics +- `JsonViewer`: Interactive JSON data viewer with property selection +- `DownloadButton`: JSON export functionality for weekly charts +- `YearlyDownloadButton`: JSON export functionality for yearly charts - `Layout`: Main layout wrapper ## Backend API Endpoints @@ -50,6 +56,7 @@ MusicCharts/ - `GET /api/health` - Health check endpoint - `GET /api/chart/dates/:year` - Get available chart dates for a year - `GET /api/chart/data/:date` - Get chart data for a specific date +- `GET /api/chart/yearly-top/:year` - Get yearly top songs ranking with calculation metrics ## Setup Instructions @@ -151,12 +158,46 @@ GET /api/chart/data/:date ] ``` +### Get Yearly Top Songs +``` +GET /api/chart/yearly-top/:year +``` + +**Parameters:** +- `year` (number): The year to get yearly top songs for + +**Response:** +```json +[ + { + "order": 1, + "title": "Luther", + "artist": "Kendrick Lamar", + "totalPoints": 245, + "highestPosition": 1, + "appearances": 8 + }, + { + "order": 2, + "title": "Die With A Smile", + "artist": "Lady Gaga", + "totalPoints": 198, + "highestPosition": 2, + "appearances": 6 + } +] +``` + ## Data Source This application scrapes data from [Music Charts Archive](https://musicchartsarchive.com), a comprehensive database of historical music charts. The scraper extracts: 1. **Chart Dates**: Available chart dates for each year 2. **Song Rankings**: Top songs with their chart positions, titles, and artists +3. **Yearly Analytics**: Multi-factor ranking algorithm calculating: + - **Total Points**: Reverse scoring (1st = 50 points, 50th = 1 point) + - **Highest Position**: Best position achieved during the year + - **Appearances**: Number of weeks on the charts ## Development @@ -166,6 +207,8 @@ This application scrapes data from [Music Charts Archive](https://musicchartsarc 2. **Backend**: Add new routes in `backend/src/routes/` and controllers in `backend/src/controllers/` 3. **Scraping**: Modify `backend/src/models/chartService.js` for new data extraction 4. **Styling**: Modify `frontend/src/styles.css` for global styles +5. **API Integration**: Update `frontend/src/services/chartApi.js` for new endpoints +6. **JSON Formatting**: Modify `frontend/src/components/JsonViewer.jsx` for custom data formats ### Testing @@ -178,17 +221,29 @@ This application scrapes data from [Music Charts Archive](https://musicchartsarc - React 18 - Custom Hooks - CSS3 with modern features -- Axios for API calls +- Axios for API calls (with extended timeouts for yearly calculations) ### Backend - Node.js - Express.js - Cheerio for HTML parsing -- Axios for web scraping +- Axios for web scraping (with optimized timeouts) - CORS for cross-origin requests - Helmet for security headers - Morgan for logging +## Performance Optimizations + +### Timeout Management +- **Frontend API Timeout**: 5 minutes (300,000ms) for yearly calculations +- **Backend Scraping Timeout**: 30 seconds per request +- **Request Delays**: 100ms between requests to respect server limits + +### Data Processing +- **Progress Logging**: Real-time progress updates during yearly calculations +- **Error Recovery**: Graceful handling of failed requests with continuation +- **Memory Optimization**: Efficient data structures for large datasets + ## Legal Notice This application is for educational and research purposes. Please respect the terms of service of the Music Charts Archive website and implement appropriate rate limiting and caching strategies for production use. diff --git a/docs/PRD.md b/docs/PRD.md index b1acf81..c0e73e4 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -71,9 +71,9 @@ GET /api/chart/data/:date - Response: Array of song objects with order, title, artist GET /api/chart/yearly-top/:year -- Purpose: Get yearly top songs ranking +- Purpose: Get yearly top songs ranking with calculation metrics - Parameters: year (number) -- Response: Array of top 50 songs with order, title, artist +- Response: Array of top 50 songs with order, title, artist, totalPoints, highestPosition, appearances ``` #### **Data Models** @@ -105,6 +105,16 @@ GET /api/chart/yearly-top/:year title: "Top Songs of 2024", songs: [Song Object Array] } + +// Enhanced Song Object (with calculation properties) +{ + order: 1, + title: "Luther", + artist: "Kendrick Lamar", + totalPoints: 245, + highestPosition: 1, + appearances: 8 +} ``` #### **Scraping Logic** @@ -143,6 +153,7 @@ App ├── DateList (Weekly mode) ├── ChartTable ├── YearlyTopSongs +├── JsonViewer ├── DownloadButton └── YearlyDownloadButton ``` @@ -156,7 +167,15 @@ App viewMode: 'weekly' | 'yearly', dates: DateObject[], chartData: SongObject[], - yearlySongs: SongObject[] + yearlySongs: SongObject[], + selectedProperties: { + position: boolean, + title: boolean, + artist: boolean, + totalPoints: boolean, + highestPosition: boolean, + appearances: boolean + } } // Loading States @@ -176,10 +195,12 @@ App #### **User Interface Requirements** - **Responsive Design**: Mobile-first approach -- **Loading States**: Visual feedback during data fetching -- **Error Handling**: User-friendly error messages +- **Loading States**: Visual feedback during data fetching with progress indicators +- **Error Handling**: User-friendly error messages with retry options - **Accessibility**: WCAG 2.1 compliance - **Cross-browser Compatibility**: Modern browser support +- **JSON Data Inspection**: Interactive JSON viewer with property selection +- **Enhanced Loading Messages**: Informative loading states for long-running operations --- @@ -286,10 +307,11 @@ class ChartAPIService { ## 📊 **Success Metrics** ### **Technical Metrics** -- **API Response Time**: < 2 seconds for chart data +- **API Response Time**: < 2 seconds for chart data, < 5 minutes for yearly calculations - **Scraping Success Rate**: > 95% successful data extraction - **Error Rate**: < 5% failed requests - **Uptime**: > 99% availability +- **Timeout Handling**: Graceful handling of long-running operations ### **User Experience Metrics** - **Page Load Time**: < 3 seconds @@ -322,6 +344,8 @@ class ChartAPIService { - **Artist Analytics**: Artist-specific statistics and rankings - **Genre Analysis**: Genre-based filtering and analysis - **Advanced Filtering**: Date range, artist, song title filters +- **Enhanced JSON Export**: Customizable data formats and property selection +- **Real-time Progress Tracking**: Visual progress indicators for long operations ### **Phase 3 Features** - **User Accounts**: Save favorite charts and analyses @@ -364,9 +388,10 @@ class ChartAPIService { ## 🎯 **Current Implementation Details** ### **Tech Stack** -- **Backend**: Node.js, Express.js, Cheerio, Axios -- **Frontend**: React 18, Custom Hooks, CSS3 +- **Backend**: Node.js, Express.js, Cheerio, Axios (with optimized timeouts) +- **Frontend**: React 18, Custom Hooks, CSS3, Enhanced JSON Viewer - **Development**: npm, nodemon +- **Performance**: Extended timeouts, request delays, progress logging ### **Project Structure** ``` @@ -395,6 +420,35 @@ m/ - `start.sh` - Startup script for both services - `README.md` - Setup and usage instructions - `PRD.md` - This product requirements document +- `docs/songList.json` - Reference JSON format for data exports +- `docs/herse-songList-export.json` - Example data export format + +--- + +## 🚀 **Recent Enhancements (Latest Update)** + +### **Enhanced Data Export System** +- **JsonViewer Component**: Interactive JSON data inspection with property selection +- **Customizable Exports**: Choose which properties to include in JSON output +- **Copy/Download Functionality**: One-click copy to clipboard and direct download +- **Format Consistency**: Matches reference format from `docs/songList.json` + +### **Performance Optimizations** +- **Extended Timeouts**: 5-minute frontend timeout for yearly calculations +- **Request Delays**: 100ms delays between scraping requests +- **Progress Logging**: Real-time progress updates during long operations +- **Error Recovery**: Graceful handling of failed requests with continuation + +### **Enhanced User Experience** +- **Property Selection**: Checkboxes to customize JSON output properties +- **Loading Indicators**: Informative loading messages for long-running operations +- **Enhanced Error Handling**: Better error messages and recovery options +- **Responsive Design**: Improved mobile and desktop experience + +### **Data Format Standards** +- **Consistent JSON Structure**: Standardized format matching industry standards +- **Calculation Properties**: Full visibility into ranking algorithm metrics +- **Flexible Export Options**: Multiple export formats for different use cases ---