Research framing
Evidence review and hypothesis workshop
Microsoft · 2024
Retention study
What makes people leave Outlook on mobile web?
The team conducted 36 interviews across six usage groups, followed by a survey with more than 300 responses. The study examined how account purpose, devices, and existing email habits affected retention.
36 interviews
6 usage cohorts
300+ survey responses
My role
Contract UX Researcherdiscussion guides, moderation, analysis, early report drafts
Timeline
2024within a Sept 2024 – Feb 2025 contract
Study lead
Khyati Sethlead researcher, with researcher and PM partners
Method
Hypothesis workshop, moderated interviews, validation survey
Project context
Outlook mobile web usage was declining faster than expected.
The product was Mini OWA: the lightweight version of Outlook opened in a phone browser rather than an installed app. Consumer usage was dropping more sharply than expected. The team needed to understand what was causing the decline and the role mobile web still played for different kinds of Outlook users.
The lead researcher and product team reviewed the questions raised by stakeholders and identified what could already be answered through existing evidence. My contribution began with the interview guides and fieldwork, after this initial framing was complete.
Stakeholder hypotheses
- Leadership, product, and engineering each had competing theories about the decline
- The workshop identified which questions were still unanswered
Existing evidence
- Data science reports and telemetry
- UX audits
- Marketing reports and prior research
Remaining questions
- What drives loyalty and churn for each kind of user?
- What role does mobile web play alongside the app and desktop?
- Early read on investments already being considered: multi-account support, home-screen shortcut, notifications
Methodology
A sequential mixed-method study across six usage cohorts
The team reviewed previous research and product data, conducted interviews, and used a survey to assess the qualitative findings with a larger sample. The final synthesis informed product recommendations and planning.
Moderated interviews
36 people across six behavior cohorts
Survey validation
300+ responses mapped to interview themes
Synthesis
Findings, opportunities, and recommendations
01
Research framing
- Purpose
- Identify which questions about declining usage had already been answered and what required additional research.
- Method
- The team reviewed data-science reports, telemetry, UX audits, marketing reports, and previous research, then held a hypothesis workshop.
- Output
- The team defined the research objectives and six cohorts based on usage across Outlook mobile web and other email access points.
02
Moderated interviews
- Purpose
- Understand what supported retention or contributed to churn, and the role mobile web played alongside apps, desktop, and third-party clients.
- Method
- We conducted remote 1:1 interviews with 36 participants across six behavior-based cohorts. Cohort-specific guides covered account purpose, devices and access points, changes in use over time, current problems, and reactions to product directions under consideration.
- Output
- The interviews explained how account purpose, email habits, and access-point choice affected mobile web usage.
- 36
- participants
- 6
- behavior cohorts
- 4–6
- people per cohort
- 50
- minutes per session
- Remote
- via UserTesting
03
Survey validation
- Purpose
- Assess whether the interview findings also appeared in a larger sample across the same behavior-based cohorts.
- Method
- The team developed a survey from the qualitative findings and collected more than 300 responses.
- Output
- The survey measured five areas identified during the interviews and showed how findings varied by cohort.
- Profile and cohortCurrent and previous use
- Email ecosystemAccounts, devices, and endpoints
- Retention and churnContinued use, switching, and abandonment
- Value and problemsPurpose, access, security, and organization
- Ideas to testMulti-account use, shortcuts, and notifications
04
Analysis and synthesis
- Purpose
- Combine the interview and survey findings before developing recommendations.
- Method
- The team analyzed account types, access-point choice, and factors that contributed to retention or churn.
- Output
- An insight tracker linked user problems and product opportunities to recommendations and planned actions.
Key findings
Account type and purpose affected where and how often people checked email.
Participants often managed several email accounts with different purposes. The study grouped these accounts into four broad types. Their importance and content affected how frequently they were checked and whether people used mobile web, an app, desktop, or a third-party client.
Primary
The account tied to daily life and important communication.
Checked frequentlySecondary
An alternate address used for selected sign-ups, services, or backup.
Used for specific tasksTertiary
An older account crowded with promotions, subscriptions, and junk.
Checked when neededAbandoned
A dormant account that still matters for an occasional recovery or reset.
Used rarelyOutlook mobile web served as a fallback within a broader email ecosystem.
- Access to older accountsEspecially Hotmail accounts that had shifted from primary to secondary or tertiary use.
- Access when apps were unavailableParticipants used mobile web when installed clients were slow, out of sync, or unavailable.
- A lightweight optionNo installation and less storage made it useful on constrained or aging phones.
Opportunity areas
Three opportunity areas from the study
The findings pointed to three ways Outlook mobile web could remain useful across different accounts and access points.
Research goalSupport continued use
Inbox organization
Help people clear accumulated junk, find critical messages, and make secondary and tertiary inboxes easier to use.
Based on accumulated content in secondary and tertiary inboxes
Notification controls
Provide contextual and selective notifications, with prioritization and focus controls.
Based on different account purposes and checking frequency
Access across endpoints
Support multiple accounts, easier return paths, and shortcuts between mobile web, Outlook apps, and third-party clients.
Based on mobile web's role as a fallback access point
Communication and impact
How the research was shared and used
Planning influence
PMs tracked findings, recommendations, and follow-up actions throughout the study. The research informed priorities for the following quarter.
Live observation
Leaders and PMs were invited to observe interviews.
Team readouts
Design, product, and engineering joined detailed working sessions.
Leadership communication
Findings were presented to leadership in India and global Outlook teams.
What I learned
What I learned about retention research
Understanding retention required looking beyond the mobile website. Participants’ older accounts, devices, installed apps, and long-term email habits also affected how they used it.
The sequence of methods worked well. Existing evidence and stakeholder hypotheses informed the research questions, interviews provided detailed explanations, and the survey was then used to validate the qualitative findings.