Signed-off-by: mbrucedogs <mbrucedogs@gmail.com>
This commit is contained in:
parent
7e5f2a16a7
commit
bcfa16aeed
63
README.md
63
README.md
@ -32,7 +32,10 @@ MusicCharts/
|
|||||||
- **Year Selection**: Choose different years to view available chart dates
|
- **Year Selection**: Choose different years to view available chart dates
|
||||||
- **Date Selection**: Browse and select specific chart dates for detailed song data
|
- **Date Selection**: Browse and select specific chart dates for detailed song data
|
||||||
- **Music Chart Display**: View ranked songs with title and artist information
|
- **Music Chart Display**: View ranked songs with title and artist information
|
||||||
- **CSV Download**: Export chart data as CSV files
|
- **Yearly Analytics**: Calculate and display top songs of the year using multi-factor ranking algorithm
|
||||||
|
- **Enhanced Data Export**: Export chart data in JSON format with customizable properties
|
||||||
|
- **JSON Viewer**: Interactive JSON data inspection with copy/download functionality
|
||||||
|
- **Property Selection**: Choose which data properties to include in exports
|
||||||
- **Responsive Design**: Mobile-friendly interface
|
- **Responsive Design**: Mobile-friendly interface
|
||||||
- **Error Handling**: Comprehensive error states and loading indicators
|
- **Error Handling**: Comprehensive error states and loading indicators
|
||||||
- **Web Scraping**: Real-time data scraping from Music Charts Archive
|
- **Web Scraping**: Real-time data scraping from Music Charts Archive
|
||||||
@ -42,7 +45,10 @@ MusicCharts/
|
|||||||
- `YearSelector`: Dropdown for year selection
|
- `YearSelector`: Dropdown for year selection
|
||||||
- `DateList`: Grid of available chart dates for selected year
|
- `DateList`: Grid of available chart dates for selected year
|
||||||
- `ChartTable`: Music chart data table with rankings
|
- `ChartTable`: Music chart data table with rankings
|
||||||
- `DownloadButton`: CSV export functionality
|
- `YearlyTopSongs`: Enhanced yearly analytics display with calculation metrics
|
||||||
|
- `JsonViewer`: Interactive JSON data viewer with property selection
|
||||||
|
- `DownloadButton`: JSON export functionality for weekly charts
|
||||||
|
- `YearlyDownloadButton`: JSON export functionality for yearly charts
|
||||||
- `Layout`: Main layout wrapper
|
- `Layout`: Main layout wrapper
|
||||||
|
|
||||||
## Backend API Endpoints
|
## Backend API Endpoints
|
||||||
@ -50,6 +56,7 @@ MusicCharts/
|
|||||||
- `GET /api/health` - Health check endpoint
|
- `GET /api/health` - Health check endpoint
|
||||||
- `GET /api/chart/dates/:year` - Get available chart dates for a year
|
- `GET /api/chart/dates/:year` - Get available chart dates for a year
|
||||||
- `GET /api/chart/data/:date` - Get chart data for a specific date
|
- `GET /api/chart/data/:date` - Get chart data for a specific date
|
||||||
|
- `GET /api/chart/yearly-top/:year` - Get yearly top songs ranking with calculation metrics
|
||||||
|
|
||||||
## Setup Instructions
|
## Setup Instructions
|
||||||
|
|
||||||
@ -151,12 +158,46 @@ GET /api/chart/data/:date
|
|||||||
]
|
]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Get Yearly Top Songs
|
||||||
|
```
|
||||||
|
GET /api/chart/yearly-top/:year
|
||||||
|
```
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
- `year` (number): The year to get yearly top songs for
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"order": 1,
|
||||||
|
"title": "Luther",
|
||||||
|
"artist": "Kendrick Lamar",
|
||||||
|
"totalPoints": 245,
|
||||||
|
"highestPosition": 1,
|
||||||
|
"appearances": 8
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"order": 2,
|
||||||
|
"title": "Die With A Smile",
|
||||||
|
"artist": "Lady Gaga",
|
||||||
|
"totalPoints": 198,
|
||||||
|
"highestPosition": 2,
|
||||||
|
"appearances": 6
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
## Data Source
|
## Data Source
|
||||||
|
|
||||||
This application scrapes data from [Music Charts Archive](https://musicchartsarchive.com), a comprehensive database of historical music charts. The scraper extracts:
|
This application scrapes data from [Music Charts Archive](https://musicchartsarchive.com), a comprehensive database of historical music charts. The scraper extracts:
|
||||||
|
|
||||||
1. **Chart Dates**: Available chart dates for each year
|
1. **Chart Dates**: Available chart dates for each year
|
||||||
2. **Song Rankings**: Top songs with their chart positions, titles, and artists
|
2. **Song Rankings**: Top songs with their chart positions, titles, and artists
|
||||||
|
3. **Yearly Analytics**: Multi-factor ranking algorithm calculating:
|
||||||
|
- **Total Points**: Reverse scoring (1st = 50 points, 50th = 1 point)
|
||||||
|
- **Highest Position**: Best position achieved during the year
|
||||||
|
- **Appearances**: Number of weeks on the charts
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
@ -166,6 +207,8 @@ This application scrapes data from [Music Charts Archive](https://musicchartsarc
|
|||||||
2. **Backend**: Add new routes in `backend/src/routes/` and controllers in `backend/src/controllers/`
|
2. **Backend**: Add new routes in `backend/src/routes/` and controllers in `backend/src/controllers/`
|
||||||
3. **Scraping**: Modify `backend/src/models/chartService.js` for new data extraction
|
3. **Scraping**: Modify `backend/src/models/chartService.js` for new data extraction
|
||||||
4. **Styling**: Modify `frontend/src/styles.css` for global styles
|
4. **Styling**: Modify `frontend/src/styles.css` for global styles
|
||||||
|
5. **API Integration**: Update `frontend/src/services/chartApi.js` for new endpoints
|
||||||
|
6. **JSON Formatting**: Modify `frontend/src/components/JsonViewer.jsx` for custom data formats
|
||||||
|
|
||||||
### Testing
|
### Testing
|
||||||
|
|
||||||
@ -178,17 +221,29 @@ This application scrapes data from [Music Charts Archive](https://musicchartsarc
|
|||||||
- React 18
|
- React 18
|
||||||
- Custom Hooks
|
- Custom Hooks
|
||||||
- CSS3 with modern features
|
- CSS3 with modern features
|
||||||
- Axios for API calls
|
- Axios for API calls (with extended timeouts for yearly calculations)
|
||||||
|
|
||||||
### Backend
|
### Backend
|
||||||
- Node.js
|
- Node.js
|
||||||
- Express.js
|
- Express.js
|
||||||
- Cheerio for HTML parsing
|
- Cheerio for HTML parsing
|
||||||
- Axios for web scraping
|
- Axios for web scraping (with optimized timeouts)
|
||||||
- CORS for cross-origin requests
|
- CORS for cross-origin requests
|
||||||
- Helmet for security headers
|
- Helmet for security headers
|
||||||
- Morgan for logging
|
- Morgan for logging
|
||||||
|
|
||||||
|
## Performance Optimizations
|
||||||
|
|
||||||
|
### Timeout Management
|
||||||
|
- **Frontend API Timeout**: 5 minutes (300,000ms) for yearly calculations
|
||||||
|
- **Backend Scraping Timeout**: 30 seconds per request
|
||||||
|
- **Request Delays**: 100ms between requests to respect server limits
|
||||||
|
|
||||||
|
### Data Processing
|
||||||
|
- **Progress Logging**: Real-time progress updates during yearly calculations
|
||||||
|
- **Error Recovery**: Graceful handling of failed requests with continuation
|
||||||
|
- **Memory Optimization**: Efficient data structures for large datasets
|
||||||
|
|
||||||
## Legal Notice
|
## Legal Notice
|
||||||
|
|
||||||
This application is for educational and research purposes. Please respect the terms of service of the Music Charts Archive website and implement appropriate rate limiting and caching strategies for production use.
|
This application is for educational and research purposes. Please respect the terms of service of the Music Charts Archive website and implement appropriate rate limiting and caching strategies for production use.
|
||||||
|
|||||||
70
docs/PRD.md
70
docs/PRD.md
@ -71,9 +71,9 @@ GET /api/chart/data/:date
|
|||||||
- Response: Array of song objects with order, title, artist
|
- Response: Array of song objects with order, title, artist
|
||||||
|
|
||||||
GET /api/chart/yearly-top/:year
|
GET /api/chart/yearly-top/:year
|
||||||
- Purpose: Get yearly top songs ranking
|
- Purpose: Get yearly top songs ranking with calculation metrics
|
||||||
- Parameters: year (number)
|
- Parameters: year (number)
|
||||||
- Response: Array of top 50 songs with order, title, artist
|
- Response: Array of top 50 songs with order, title, artist, totalPoints, highestPosition, appearances
|
||||||
```
|
```
|
||||||
|
|
||||||
#### **Data Models**
|
#### **Data Models**
|
||||||
@ -105,6 +105,16 @@ GET /api/chart/yearly-top/:year
|
|||||||
title: "Top Songs of 2024",
|
title: "Top Songs of 2024",
|
||||||
songs: [Song Object Array]
|
songs: [Song Object Array]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Enhanced Song Object (with calculation properties)
|
||||||
|
{
|
||||||
|
order: 1,
|
||||||
|
title: "Luther",
|
||||||
|
artist: "Kendrick Lamar",
|
||||||
|
totalPoints: 245,
|
||||||
|
highestPosition: 1,
|
||||||
|
appearances: 8
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
#### **Scraping Logic**
|
#### **Scraping Logic**
|
||||||
@ -143,6 +153,7 @@ App
|
|||||||
├── DateList (Weekly mode)
|
├── DateList (Weekly mode)
|
||||||
├── ChartTable
|
├── ChartTable
|
||||||
├── YearlyTopSongs
|
├── YearlyTopSongs
|
||||||
|
├── JsonViewer
|
||||||
├── DownloadButton
|
├── DownloadButton
|
||||||
└── YearlyDownloadButton
|
└── YearlyDownloadButton
|
||||||
```
|
```
|
||||||
@ -156,7 +167,15 @@ App
|
|||||||
viewMode: 'weekly' | 'yearly',
|
viewMode: 'weekly' | 'yearly',
|
||||||
dates: DateObject[],
|
dates: DateObject[],
|
||||||
chartData: SongObject[],
|
chartData: SongObject[],
|
||||||
yearlySongs: SongObject[]
|
yearlySongs: SongObject[],
|
||||||
|
selectedProperties: {
|
||||||
|
position: boolean,
|
||||||
|
title: boolean,
|
||||||
|
artist: boolean,
|
||||||
|
totalPoints: boolean,
|
||||||
|
highestPosition: boolean,
|
||||||
|
appearances: boolean
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Loading States
|
// Loading States
|
||||||
@ -176,10 +195,12 @@ App
|
|||||||
|
|
||||||
#### **User Interface Requirements**
|
#### **User Interface Requirements**
|
||||||
- **Responsive Design**: Mobile-first approach
|
- **Responsive Design**: Mobile-first approach
|
||||||
- **Loading States**: Visual feedback during data fetching
|
- **Loading States**: Visual feedback during data fetching with progress indicators
|
||||||
- **Error Handling**: User-friendly error messages
|
- **Error Handling**: User-friendly error messages with retry options
|
||||||
- **Accessibility**: WCAG 2.1 compliance
|
- **Accessibility**: WCAG 2.1 compliance
|
||||||
- **Cross-browser Compatibility**: Modern browser support
|
- **Cross-browser Compatibility**: Modern browser support
|
||||||
|
- **JSON Data Inspection**: Interactive JSON viewer with property selection
|
||||||
|
- **Enhanced Loading Messages**: Informative loading states for long-running operations
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@ -286,10 +307,11 @@ class ChartAPIService {
|
|||||||
## 📊 **Success Metrics**
|
## 📊 **Success Metrics**
|
||||||
|
|
||||||
### **Technical Metrics**
|
### **Technical Metrics**
|
||||||
- **API Response Time**: < 2 seconds for chart data
|
- **API Response Time**: < 2 seconds for chart data, < 5 minutes for yearly calculations
|
||||||
- **Scraping Success Rate**: > 95% successful data extraction
|
- **Scraping Success Rate**: > 95% successful data extraction
|
||||||
- **Error Rate**: < 5% failed requests
|
- **Error Rate**: < 5% failed requests
|
||||||
- **Uptime**: > 99% availability
|
- **Uptime**: > 99% availability
|
||||||
|
- **Timeout Handling**: Graceful handling of long-running operations
|
||||||
|
|
||||||
### **User Experience Metrics**
|
### **User Experience Metrics**
|
||||||
- **Page Load Time**: < 3 seconds
|
- **Page Load Time**: < 3 seconds
|
||||||
@ -322,6 +344,8 @@ class ChartAPIService {
|
|||||||
- **Artist Analytics**: Artist-specific statistics and rankings
|
- **Artist Analytics**: Artist-specific statistics and rankings
|
||||||
- **Genre Analysis**: Genre-based filtering and analysis
|
- **Genre Analysis**: Genre-based filtering and analysis
|
||||||
- **Advanced Filtering**: Date range, artist, song title filters
|
- **Advanced Filtering**: Date range, artist, song title filters
|
||||||
|
- **Enhanced JSON Export**: Customizable data formats and property selection
|
||||||
|
- **Real-time Progress Tracking**: Visual progress indicators for long operations
|
||||||
|
|
||||||
### **Phase 3 Features**
|
### **Phase 3 Features**
|
||||||
- **User Accounts**: Save favorite charts and analyses
|
- **User Accounts**: Save favorite charts and analyses
|
||||||
@ -364,9 +388,10 @@ class ChartAPIService {
|
|||||||
## 🎯 **Current Implementation Details**
|
## 🎯 **Current Implementation Details**
|
||||||
|
|
||||||
### **Tech Stack**
|
### **Tech Stack**
|
||||||
- **Backend**: Node.js, Express.js, Cheerio, Axios
|
- **Backend**: Node.js, Express.js, Cheerio, Axios (with optimized timeouts)
|
||||||
- **Frontend**: React 18, Custom Hooks, CSS3
|
- **Frontend**: React 18, Custom Hooks, CSS3, Enhanced JSON Viewer
|
||||||
- **Development**: npm, nodemon
|
- **Development**: npm, nodemon
|
||||||
|
- **Performance**: Extended timeouts, request delays, progress logging
|
||||||
|
|
||||||
### **Project Structure**
|
### **Project Structure**
|
||||||
```
|
```
|
||||||
@ -395,6 +420,35 @@ m/
|
|||||||
- `start.sh` - Startup script for both services
|
- `start.sh` - Startup script for both services
|
||||||
- `README.md` - Setup and usage instructions
|
- `README.md` - Setup and usage instructions
|
||||||
- `PRD.md` - This product requirements document
|
- `PRD.md` - This product requirements document
|
||||||
|
- `docs/songList.json` - Reference JSON format for data exports
|
||||||
|
- `docs/herse-songList-export.json` - Example data export format
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 **Recent Enhancements (Latest Update)**
|
||||||
|
|
||||||
|
### **Enhanced Data Export System**
|
||||||
|
- **JsonViewer Component**: Interactive JSON data inspection with property selection
|
||||||
|
- **Customizable Exports**: Choose which properties to include in JSON output
|
||||||
|
- **Copy/Download Functionality**: One-click copy to clipboard and direct download
|
||||||
|
- **Format Consistency**: Matches reference format from `docs/songList.json`
|
||||||
|
|
||||||
|
### **Performance Optimizations**
|
||||||
|
- **Extended Timeouts**: 5-minute frontend timeout for yearly calculations
|
||||||
|
- **Request Delays**: 100ms delays between scraping requests
|
||||||
|
- **Progress Logging**: Real-time progress updates during long operations
|
||||||
|
- **Error Recovery**: Graceful handling of failed requests with continuation
|
||||||
|
|
||||||
|
### **Enhanced User Experience**
|
||||||
|
- **Property Selection**: Checkboxes to customize JSON output properties
|
||||||
|
- **Loading Indicators**: Informative loading messages for long-running operations
|
||||||
|
- **Enhanced Error Handling**: Better error messages and recovery options
|
||||||
|
- **Responsive Design**: Improved mobile and desktop experience
|
||||||
|
|
||||||
|
### **Data Format Standards**
|
||||||
|
- **Consistent JSON Structure**: Standardized format matching industry standards
|
||||||
|
- **Calculation Properties**: Full visibility into ranking algorithm metrics
|
||||||
|
- **Flexible Export Options**: Multiple export formats for different use cases
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user