Content Customizer at Headspace
Introduction:
The Content Customizer is Headspace’s innovative personalized recommendation system. Designed to enhance user experience, this system utilizes historical data to train its machine learning model, providing users with content tailored to their preferences and past interactions.
Objective:
The primary goal was to evaluate the effectiveness of the recommendation system as a whole, ensuring its efficiency before deploying it to production. Online Controlled Experiments, particularly A/B testing, were employed as the gold standard for estimating causality with high probability.
Experiment Design:
The experiment encompassed various components:
– Recommendation Surface Area: Three main areas were identified for the experiment, including the Today tab, the Hero module, and the recommended sub-tabs.
– Platform Selection: The experiment was conducted on both Android and iOS platforms.
– Audience Selection Criteria: The machine learning models were trained on historical data, ensuring the distribution of characteristics in the training set matched that during inference.
– Driver Metrics and Business KPIs: Metrics such as Content Start, Content Complete, and Unique Content Start were used to measure the impact of the ML model. These metrics were closely aligned with business KPIs like Daily/Weekly/Monthly active users and churn rate.
Architecture:
The Content Customizer model uses a robust architecture for experiments, which includes:
– Content Customizer Model: Generates personalized recommendations for every eligible user.
– Prediction Service: Sends predictions to the layout service asynchronously.
– Layout Service: Creates a layout for the client for respective tabs.
– Optimizely: An online A/B testing platform that administers the online controlled experiment.
Evaluation:
The evaluation of A/B testing was divided into two parts: Statistical Significance and Practical Significance. The evaluation considered factors like effect size, p-value, confidence interval, and practical significance thresholds.
Impact of Rohan’s Work:
Rohan Singh Rajput’s leadership and expertise were instrumental in the success of the Content Customizer project. His deep understanding of machine learning, combined with his passion for data-driven innovation, ensured that the project not only met but exceeded its objectives. Rohan’s approach to integrating experimentation into the process showcased the profound impact of marrying technology with user-centric design. His contributions have set a new benchmark for user engagement at Headspace, underscoring the transformative power of AI in enhancing user experience.
Conclusion:
The Content Customizer project, under Rohan’s guidance, has set a precedent for how AI can be effectively integrated into platforms like Headspace to drive user engagement and satisfaction. The project’s success is a testament to Rohan’s expertise and the potential of AI-driven solutions in the digital wellness space.
