Amazon NPS Survey Generator
Allowing businesses to get insights across Amazon through building surveys
Context
I joined Amazon’s Customer Experience Benchmarking team in June 2022 and have only worked in the e-commerce and customer-facing product world. I was interested in taking on a challenge of learning how to design for internal tools. Designing for external vs internal customers isn’t actually too different, in the end my goal is to prioritize the customer experience and offer a product that is delightful, helps a user achieve their goals and makes them wanting to come back to use it.
Existing Problems
Lack of standardization between organizations
Amazon has many internal organizations utilizing NPS surveys for their business purposes, such as AWS, shopping, grocery, pharmacy etc. These surveys all go through the Research Excellency Group(REG) to make sure these surveys meet research standards and are written according to guidelines. However the lack of a single source of truth leads to more manual work for REG and businesses being unsure of the right way to write surveys.
Unnecessary churn between stakeholders
Each study averages a survey cost of $48K paid to third-party vendors, 5 weeks of manual effort across PCIM, A&A and Research Excellence per study; equates to $3.8MM in effort that could be automated each year. Surveys are handled through excel files that are sent back and forth by email, with no transparency on their status and they must go through multiple rounds of approvals.
Human error leading to mistakes in analysis
Mistakes in the questionnaire writing and survey programming would lead to analysis errors, resulting in frustration for all parties involved. Results and data also has to be manually inputted and formatted. Business owners will occasionally want to cross-compare to other geographies or domains, but when surveys are not standardized they cannot quickly analyze the data.
Goals
Offer a single source of truth
With the Research Excellency Group having to approve NPS surveys to make sure they are unbiased and correctly formatted, this causes more manual work for REG. By having a survey templates, it can make writing surveys for business owners more accessible, reduce churn and make data analysis easier.
Increase adoption rates
Current regular adoption rates among Product Consumer Insight Managers(PCIMs) in Amazon was about 12%. Although 60% of PCIMs mentioned being exposed to the tool, they found the experience difficult to use, causing them to prefer other methods to do their work of designing surveys and analyzing data.
Fit for all use cases across Amazon
Surveys need to not only be able to support different domains, but needs to support more than 100+ countries and 15+ languages for it’s NPS surveys. With cultural and language differences and a lack of understanding of these nuances, it needs to be ensured that the surveys can be fielded for all of these different use cases.
Understanding our Users
An investigation had to be done on how our main users(PCIMs) currently use Icarus for their workflow, and what were their pain points when using the process to figure out where process improvements could be made. The below journey map shows all the steps a PCIM must go through to complete a survey, and where they use Icarus or other offline tools. Data was collected after an individual hour long interview with all 8 PCIMs for an hour to discuss their process and workflow and where they were having frustrations. Commonalities have been organized and steps in the process were grouped together in 5 steps; Scoping, Questionnaire Design, Finalizations + Fielding, Data Processing and reporting. Based on PCIM feedback, a few ideas were brainstormed on where improvements could be made. Most improvements could be made at the questionnaire design phase and improvements would affect the downstream, hence the choice to focus on the questionnaire designer.
Current process journey map
Ideation
Priortizing Features
UX designers have to balance timing, feasibility and impact when designing to features for an existing tool. In order to balance frugality with thinking big, I created a “Bang vs Buck” chart where we could figure out what features would require most work or less work, and what would be highest impact for users and lowest. The improvement features were categorized into what step of the survey process they belong to, with scoping, creation, fielding and Analysis + Reporting. The majority of feature requests where found in the “Creation” phase and it was decided that it would be best to focus on the survey creation phase, as that would also affect the survey’s downstream and analysis and reporting.
Design
Planning
I created a user flow diagram to match the business’s process to the proposed experience after prioritize which features would have the most impact for users.
Competitive Analysis
The team at the time was used to Qualtrics but wanted to move away from it due to requiring more standardization across Amazon domain and preference to keep all data internal. I sought out other tools Product Consumer Insights Managers have used in the past and took notes of what was working well, and what wasn’t.
SurveyMonkey balances user friendliness and custom options. It offers templates and a question bank UI that essentially works like a shopping cart, so users can pick and choose questions that would fit their survey.
Google Forms offers a very simple UI for creating short simple surveys. It is not able to capture all the advanced use cases needed, such as looping questions.
Qualtrics can offer anything a survey creator can dream of, including looping questions, question logic. However on the flip side, it has a bit of a learning curve for all the intricacies it offers.
Features
Quiz Experience
In order to decide what kind of survey is suitable for their use case, a business user will have a 30 minute - 1 hour meeting with the Research team to determine what kind of survey and how they should field it. I worked with REG to simplify this questionnaire and see how we could simplify the path without missing important questions.
Initial mocks
Final design mocks
Survey Designer
Early wireframe before a design style was chosen, I decided to quickly make style-agnostic mocks so we could get early feedback on the interactions first before diving deep into the visual style.
After an existing AWS design style was decided on for this product, I remade the early mocks along with incorporating user feedback and new requirements, such as a warning if users edit “REG Approved” questions and added an easy way to navigate through the survey(since they can be quite long). I also added an estimate on how long a survey would take to complete.
