Multi-Class Sentiment Classification of Product Reviews Using Classical NLP Models
How well can classical NLP pipelines predict the 1–5 star rating of a product review from its text alone, when the rating classes are heavily imbalanced?
- Data
- The first 50,000 camera reviews from the Amazon US Customer Reviews dataset. The review headline and body were combined, with the headline repeated to give it more weight, and HTML line breaks were removed.
- Target
- Star rating (five classes). The distribution is strongly skewed: 31,516 five-star reviews against 2,382 two-star reviews.
- Methods
- Text preprocessing: lower-casing, contraction expansion, punctuation removal and normalisation of repeated letters (e.g. “soooo” → “soo”)
- TF–IDF features inside scikit-learn pipelines
- Hyperparameter search over TF–IDF and model settings, optimising macro F1
- Feature engineering
- Headline and body concatenated, with the headline duplicated to up-weight it
- Removal of HTML line-break artefacts found in more than 5,000 reviews
- Unigram and bigram TF–IDF features, with vocabulary size and document-frequency cut-offs tuned
- Evaluation design
- Stratified 80/20 train–test split (10,000 test reviews). Grid search (randomised search for XGBoost) with 5-fold cross-validation on the training set; final scores reported on the held-out test set.
- Models compared
- Multinomial Naive Bayes
- Logistic Regression
- Linear SVM
- Random Forest
- XGBoost
- Metrics
- Accuracy
- Precision
- Recall
- Weighted F1
- Macro F1
- Per-class F1
- Result
- Tuned Logistic Regression reached the highest macro F1 (0.655) and was selected. Linear SVM scored similarly overall but was weaker on the two- and three-star classes. Per-class F1 for the selected model ranged from 0.91 for five-star reviews to 0.47 for two-star reviews, so the middle ratings remain the hard cases.
- My contribution
- Individual project: data preparation, exploratory analysis, pipeline design, model tuning and evaluation.
- Limitations
- Severe class imbalance: minority star ratings are predicted much less reliably than five-star reviews.
- A single product category and the first 50,000 rows of the source file.
- Star ratings are an imperfect proxy for the sentiment expressed in the text.
- Possible next steps
- Test lemmatisation and emoji handling, which the notebook lists as further tuning steps.
- Evaluate formulations that treat star ratings as ordered rather than as unrelated classes.
- Research relevance
- Extends the undergraduate opinion-mining thesis from a hand-built three-class Naive Bayes to a systematic, cross-validated comparison of five classifiers on a larger and imbalanced dataset.
- Tools
- Python
- Pandas
- NumPy
- NLTK
- scikit-learn
- XGBoost
- Matplotlib
- Seaborn
- Jupyter