Algorithm Testing Feature

EVERGAGE

Collaboration + Communication | Leadership + Initiative | Strategic Impact | User Centered UX Design 

laptop-recipe simulator

Project Description

The Company: Evergage, (now SaleForce Interaction Studio) was a cloud based B2B personalization and customer data platform that used data and machine learning to provide cross channel (web, mobile, email, and third party) personalized business solutions to their clients.

Algorithm Testing Feature: A tool that allowed the user to create and test custom machine learning algorithms that outputed personalized recommendations for targeted users.

These algorithms were used in email and web campaigns and it was imperative that they were thoroughly tested before using them live in a campaign.

The capabilities and client demands drastically increased and the tool's functionalities and interface didn't change. 

Users:

  1. VIP Clients using Evergage for complex algorithms.

  2. Standard Clients using Evergage for basic or "out of the box" algorithms. 

Problem: The process to test the algorithms was all manual and most of the complex testing was done outside of the platform using internal tools and work arounds. In order for a large prospect to convert, this process needed to be automatic and in the platform. 

Goal: To automate the algorithm testing process by adding advanced functionality to Evergage’s algorithm testing tool (AKA Recipe Simulator Feature).

Hypothetical Measure of Success:

  1. Reduced number of errors in campaigns using the algorithms
  2. NPR survey for client satisfaction 
  3. Reduced time it takes for a client to test an algorithm

Impact

  1. The above mentioned prospect signed (huge financial impact).
  2. Support, Internal Data Scientists, + Clients confirmed the goal of creating a more effective and deeper testing deeper was achieved. 
  3. Verbal feedback suggested that the feature was useful for current clients and internal support teams.
  4. Verbal feedback from the Customer Success team and Director of Learning that there was a boost in moral because they felt prepared and in the loop.
  5. The Sales Team reported that the new functionalities and UI made it easier for them to showcase Evergage's value to prospects.

Context: Evergage provides advanced machine learning for businesses to recommend products, categories, and other content to personalize their visitors’ experiences. The recommendations can be customized with a variety of settings such as trending, show similar items, user affinities, or specific brands or categories through the creation of “recipes.” The recipes are added into our campaigns to personalize recommendations. Recipes must be tested to ensure they were set up properly and working correctly.

My Role
Product Designer, Project Lead

The Team
Myself, 2 Engineers, CTO, UX Director, Data Scientists, Director of Learning and Development.

Methods
Interaction design, user flows, collaboration, initiative, leadership, visual design

Tools
Axure RP, White board, Evergage, Jira, Google Docs

Feature Before

Old recipe testing – marked up

Feature After

Recipe Simulator

How it Started

The wireframes and functionality was set by the CTO, executive team, and Engineers. They had created a similar functionality outside of the platform and had a vision of how it would generally look in the platform. The CTO white boarded his solution in our weekly design meeting.

The UX Director, who's strength was visual design, was assigned the project and transferred the CTO's white board sketches verbatim into Axure RP. During our team design reviews I had provided feedback on the designs multiple times that the main user flows were not accounted for in the new designs. To my knowledge, the feature was only internal so despite knowing the UX problems, the internal team understood how the feature worked so it wouldn't be a problem. Also, she was the lead designer and my voice only went so far in this situation. 

The designs were handed off to the two engineers to start building. The UX Director went on vacation and I was temporily the design lead for the project while she was away.

Note - the UX Director, CTO, and I work very closely together and are all present during the design reviews. 

Huston, we have a problem!

The engineers noticed the major gaps in the feature and asked me to clarify. I suggested a few band-aids and mentioned that to truely clarify, we'd have to go back to the whiteboard which would take a ton of time that neither of us had. I was focused on current high impact projects and this project was due yesterday (#startup life). It would have to be fine because it was only internal users. 

Plot twist - I was informed that this feature would in fact be released to ALL users, internal and external. 

At that point, I used Evergage on Evergage data to check the screen usage. It turns out that 50 users regularly used the screen and a portion of them were important users. I went to the CTO and highlighted the major UX problems, the usage data, and proposed a new plan that would add a week to the timeline and move me from my other projects to focus on this one. 

With the engineers we discussed technical implications, business impact, and user impact. He approved my request. 

Note - this purpose of this project is to showcase my work and strengths and is no way intended to reflect negatively on anyone else. 

Initial Designs (Dropdown Closed)

Dropdown closed w.o nav

Initial Designs (Dropdown Open)

Dropdown open w.o nav

Ideation + Whiteboarding

The engineers and I went back to the whiteboard. This was a highly interactive screen with new concepts and behaviors. There were some major structural requirements and concepts that weren’t defined, which left holes in the design and the experience. For example, the relationship between a test group, adding a user, and the recipe. In addition to showing it in the UI, I helped refine the feature and how it worked. 

We collaborated and wireframed a modified solution that would account for the user flows, interactions, and define the relationship between the new concepts. The engineers were very helpful to address the technical limitations and I helped them understand the user requirements. 

This is a new concept and it's critical that the user has the support to understand how to use the feature. We determined what needs to be intuitive and then added support including helper text, callouts with more information, and support docs.

Examples of Use cases / Interaction 

  1. How do you add / edit / delete a rule
  2. How do you add / edit / delete a test group
  3. How do you create your first algorithm
  4. What does it look like when nothing is created?

Marked Up to Show the Changes

Dropdown open w.o nav -marked up

DOCUMENTED THE FEATURE & ADDED ORGANIZATION

I created a working feature requirement doc in Google docs for the team which contained all the requirements, definitions, questions, user flows, and eventually the burn down list. It was essential to document and stay on the same page, especially because we were on a tight time schedule. The three of us met for an additional daily team scrum to sync up.

Updated Designs 

I modified the design files created by the UX Director in Axure to reflect the new wireframes. The clean files allowed me to quickly make the modifications. 

I worked on my design check list and improved the visual design. This included checking that the elements and styles were consistent, i.e the “add user” icon was the same for both random and specific user, as well as check the hierarchy and spacing. Then moved to the smaller interactions like the different hover states, deleting a user, deselecting a filter, etc. Highlighted in the designs are some of the specific interaction areas I focused on and enhanced.

Recipe-builder-part-1 with notes
Recipe Simulator with notes

Implementation Collaboration

I handed off the designs to the two engineers. One focused on the back end and the other on the front end of the feature. I worked closely with both, especially the front end dev. This was a very interactive screen and with little time I had to hand over the designs with enough of the interactions accounted for. While they were building I was working through some of the known outstanding interaction questions, and was available for support as they discovered new interaction issues.

The Feature Release Plan

This was an impactful feature and a drastic change which meant that the Customer Success team really needed to prepare to talk to their clients and the users needed to have support. Communicating releases has been a pain point between the departments and is something I’ve worked on improving.

For this project I took advantage of leading the project and implemented a successful release plan. This included working with the the Director of Learning to produce adequate support documentation, communicating with the Customer Success through email and quick updates in team meetings to keep them apprised of the changes and the shifting release time line, and working with the CTO to add an informational tour and in app message explaining the feature.

Helper Call out Message
static1.squarespace