Connection Catalog
Validation Study
STREAMSETS
Project Description
The Company: StreamSets is a B2B DataOps platform that enables businesses to quickly transport continuous data with quality so they can confidently use their data.
The feature: Connection Catalog. This feature was heavily requested and intended to add functionality to solve a significant bottleneck in the users’ workflow. It was so critical that each department was involved.
The purpose of the research study was to validate the team’s hypotheses we had about this new feature.
Impact of the Study The feature is critical to improving usability and solving for major bottlenecks in the users' workflow that were directly and negatively impacted sales deals and renewals. The study validated the team’s hypotheses and gave insight into critical product decisions that helped the team move forward with the feature.
My Role
Lead UX Researcher
The Team
Product Manager, UX Team (Myself and the UX Lead), CPO, and Customer Successs Team
Methods
Survey development, Facilitation, Semi Structured Interview
Tools
Google Form, Zoom, Figma, Youtube
The Story
Step 1 | Define
Defining the Requirements
My team collaborated with the Product Manager to identify the requirements for the study and the feature. The Product Manager defined the feature goals and problem definition, while my team defined the logistics of the study, the stakeholders, and the target users/personas. We worked together to develop the hypotheses.
Context: StreamSets allows companies to move data by creating pipelines. Each pipeline consists of many data connections that deliver specific types of data through each pipeline.
Problems
#1 Problem: Data security and privacy is very important to the users and their companies. A common and crucial task in moving data is creating data connections, which require credentials. The Operator persona was the only user with permissions to the credentials thus the only users able to create connections. This caused a bottleneck in the user flow because they weren’t able to delegate the task to the Data Engineer, who should be creating them.
Goal: To save time for the Operator persona and create a more efficient user flow by allowing the Data Engineer to create connections without the need for credentials.
#2 Problem: Creating connections became a redundant task because the users were spending a lot of time creating the same connections for different pipelines.
Goal: To save the user time by allowing connections to be reusable.
#3 Problem: Managing connections was difficult and time consuming because there wasn’t a central location for all the connections.
Goal: To save the user time by creating a central location for the user to create and manage their connections.
Step 2 | Set up
Preparing for the Study
Find the users: Since the product is B2B I worked with the Customer Success team to find 5 users for the study. I shared the feature and study information with the team and explained the type of users we needed and what we needed from them.
The Script: I created a survey using Google Form that contained a mix of questions and potential designs and workflows. I used the study and feature goals as well as best practices as my guidelines while I developed the survey. In the beginning I asked questions about security to gain insights into the role of security at their company. I added questions to understand the “why” behind the users’ reactions. For example,
“What value do you see this feature bringing to you and your team?”
“If you could add or change anything about the feature what would it be and why?”
“What are you most excited about this feature?”
The Format: Each test was a 45 minutes long video call. I shared my screen and walked through the survey with the user and filled in their answers in real time. This was an efficient way to accurately capture the data.
Step 3 | The Study
Study Protocol
Purpose of study: To validate our two hypotheses and to answer a few questions about the design solutions.
Method: User Interview over video call (Zoom)
Timeline: 1 week
Users: 5
Hypotheses
1) Saved connections will help reduce the security concern and allow users to create and share connections without the need to share credentials.
2) Creating reusable and shareable connections will increase teams productivity and improve the users’ experience.
Questions
- How does each company view and handle security?
- Which option for creating a connection is best - using a dropdown or right in the connection?
- Are there any usability concerns outside of the hypotheses?
The goal for our team was to use the data from our user research to inform our product design with validated evidence. We used the method of validated research meaning that in qualitative research, 4-6 consistent responses are needed for validation.
The Validation Key
Happenstance - One users has given feedback
Coincidence - Two users have given the same feedback
Evidence - Three users have given the same feedback.
Validated Evidence - 4-6 users have given the same feedback.
Step 4 | Results & Findings
A Successful Study
Hypothesis 1 | Usability and Productivity (5/5 confirmed)
Saved connections will help reduce the security concern and allow users to create and share connections without the need to share credentials.
Notable Findings About Security Not everyone uses a vault to store their credentials, which led us to think about additional design solutions that don’t involve a vault.
Hypothesis 2 (5/5 confirmed)
Creating reusable and shareable connections will increase teams productivity and improve the users’ experience.
Notable Findings About Usability. One user expressed their user pain, "This will improve our delivery capacity because adding credentials everywhere is very painful and we have a lot of JDBC stages. When asked about the options for creating connections 1 of the users preferred creating the connection right in the pipeline while 2 users prefer to use the dropdown.
Powerful User Quotes
“One stop for all connection management”
"I'm very excited about this feature"
“It will increase productivity and decrease errors for sure.”
“The sooner we get it the better. It's a great enhancement to the tool”
“A good value, something for sure will use a lot”
Retrospective
Overall the study went very well. The team got the information and feedback we needed to move forward with the feature. After the study I created a powerpoint with the results and shared it with the concerned teams.
If I had more time to do the study I would have liked to find a 6th person to add to the validity of the study. In my ideal scenario I would like to conduct these user studies in person to see the users' reactions to the designs. This would give me a more genuine insights into their opinions.