Overview
TLDR
Melatone is a real-time facial skin tone classification and palette recommendation system. The pipeline combines face localization, skin region segmentation, tone classification, and rule-based palette mapping in a Streamlit interface.
Melatone is a computer vision pipeline for estimating facial skin tone and mapping the result to a curated color palette. The system accepts either uploaded images or webcam input, then performs preprocessing, inference, and palette rendering in a single interactive workflow.
The technical challenge is not just classification accuracy. The harder part is making the system stable under uncontrolled lighting, camera noise, and partial occlusion. Those conditions make the task closer to a real deployment problem than a clean benchmark exercise.
Pipeline
Melatone is organized as a sequence of modular stages rather than a single end-to-end model. The main components are face detection, skin region extraction, baseline classification, deep learning classification, and palette generation.
Face and skin segmentation
The first stage uses OpenCV’s Haar Cascade classifier for face localization. I used this approach because it is lightweight, fast, and sufficient for bounding the facial region before more expensive processing.
After detection, the image is converted to HSV space and thresholded with predefined skin-tone ranges. HSV was a better fit than RGB because it separates chromatic information from brightness, which makes the segmentation less sensitive to lighting variation. The resulting mask is cleaned with morphological opening and closing, then smoothed with Gaussian blur to reduce noise.
The final region of interest is extracted with bitwise masking and heuristic cropping around the cheek and forehead. This gives a compact skin sample for classification while avoiding full-face background contamination.
Baseline classifiers
Before deep learning, I tested HOG features with KNN and XGBoost. These baselines were useful because they established a lower bound on performance and exposed the limits of hand-crafted features for this task.
Both models struggled under changes in illumination and skin-tone variation. Their accuracy ranged from roughly 30% to 71%, depending on the configuration, which showed that the problem needed a more expressive classifier.
Deep learning classifier
The final classifier was trained on 1,500 labeled images across three classes: Light, Medium, and Dark. I compared several architectures, including VGG16 and EfficientNet variants, before settling on a custom EfficientNet-based model in PyTorch.
VGG16 produced weaker generalization, even with training refinements such as early stopping. EfficientNet performed better because its architecture offered a stronger tradeoff between representational capacity and parameter efficiency. The final model reached about 75.3% validation accuracy.
Training used Adam optimization, dropout regularization, and early stopping. These choices improved stability during optimization and reduced overfitting relative to the earlier baselines.
What I used
Haar Cascade
Lightweight face detection for early region localization before heavier processing.
HSV Color Space
Separates brightness from color information, which made skin masking noticeably more stable.
Gaussian Blur
Smoothed out noisy segmentation artifacts and unstable pixel regions before sampling.
Kalman Filter
A useful lens for thinking about how to smooth observations across time in live video systems.
HOG + XGBoost / KNN
Classical baselines I built first. They were helpful for understanding the problem before moving to CNNs.
EfficientNet
The final deep learning classifier, which generalized considerably better than earlier approaches.
Evaluation
I added a batch evaluation module to compare KNN, XGBoost, and EfficientNet predictions on multiple captured images. This made it easier to inspect model bias and identify where predictions diverged across classes.
The main error pattern was class imbalance. The Light class showed more false positives than the other categories, while Medium and Dark were more stable. Confusion matrix analysis confirmed that the model was most reliable when the image conditions were consistent and the skin region was clearly visible.
White balance correction was added later using a gray-world algorithm. This preprocessing step reduced color drift caused by warm indoor lighting and cool outdoor lighting, and it improved consistency across webcam captures and uploaded images.
Interface
The interface was built with Streamlit so the system could support both file upload and live webcam capture. The original version used MediaPipe hand gestures to trigger capture, but user testing showed that a simple keypress workflow was more reliable across devices and much easier to reproduce.
That change improved latency and reduced interaction overhead without affecting the core pipeline. The interface displays the predicted skin-tone class, the palette recommendation, and the rendered HEX swatches in one screen.
Recommendation logic
The recommendation module uses deterministic class-to-palette mapping. Each tone category is linked to a curated set of colors inspired by seasonal color theory and adapted for Southeast Asian skin tones.
This design keeps the output explainable. Instead of learning the recommendation end-to-end, the model is only responsible for classification, while the application handles palette selection as a transparent post-processing step.
Results
The final system runs in real time at roughly 2-3 frames per second on a standard laptop. Among the tested approaches, the custom EfficientNet model outperformed the classical baselines and the earlier VGG16 experiments.
HOG + KNN reached about 63% accuracy, and HOG + XGBoost reached about 71%. The final EfficientNet model reached about 75.3% validation accuracy and produced the most balanced predictions across classes.
Limitations
The dataset contains only 1,500 images, which limits both generalization and fairness across skin tones, ethnicities, and lighting conditions. The current three-class taxonomy is also coarse, since real skin-tone variation is continuous rather than discrete.
A stronger next version would use a larger and more diverse dataset, finer-grained labels, and a more robust detector such as YOLO. It would also benefit from confidence visualization and more detailed error analysis.
Conclusion
Melatone is a complete computer vision pipeline for skin-tone estimation and palette recommendation. It combines classical preprocessing, deep learning classification, deterministic mapping, and a real-time interface into a deployable system.
The project showed that useful vision systems depend on more than model accuracy. Preprocessing quality, lighting robustness, interface design, and evaluation discipline were just as important as the classifier itself.