Notification Center

An A/B usability experiment to help prioritize a much-needed feature.

After beginning the Student Loan Match project at a leading FinTech benefits organization, it soon became clear that the number of in-platform communications necessary for this new program type would be too high to use existing components. Participants were already collecting "banner bloat" within our platform without any way to dismiss the messages because we had not yet built the idea of "sessions" into our backend. I therefore created an A/B, between-subjects study to determine if the addition of a notification center would make a significant difference on the participant experience and, if so, which notification center concept was most usable.

I first began the design process with an audit of our existing in-platform communications, as well as the communications we would need to send in the future for participants in Student Loan Match programs. Additionally, I conducted competitive research on a variety of financial and non-financial platforms to understand how other companies were handling notification systems. From my research, I developed three Notification Center approaches that I translated into low-fidelity concepts. I presented these low-fidelity concepts at a UX peer review to gather general and heuristic feedback on the designs, which I then used to inform tweaks to the designs.

I then translated these low-fidelity designs into high-fidelity prototypes to be used in an A/B test, utilizing a relatively complex program communication as the "message" for participants. Since I wanted to investigate whether a notification center would be a significant improvement from using our existing components, I also created a high-fidelity "control" prototype that I could analyze the three notification center concepts against. I then defined the aspects of the experience we were trying to test:

    Discoverability
    Ease of use
    Efficiency
    Accuracy
    Qualitative reaction/reflection
    Information retention

I also defined the measures that I planned to collect, beyond qualitative feedback from think-aloud protocol: Number of errors, time to complete each task successfully, right/wrong answers (binary), and System Usability Scale ratings. Since many of the usability aspects I was looking to test were predicated on the assumption of a user seeing a certain communication for the first time, I determined the appropriate experimental setup to be between-subjects. I included all of this information in my formal test plan.

Due to time and resource constraints, I organized the tests to be unmoderated through the UserTesting.com platform. I screened for users that were employed full- or part-time by someone other than themselves, who have student loans or have had them in the past. I then analyzed the results using frequency counts, medians, and nonparametric inferential statistics (namely, Wilcoxon Rank Sum and Mann-Whitney U Tests). I utilized the R programming language combined with R Studio for my analysis.

Based on my results, I was able to support my hypothesis that, for the intended application with Match participants, a notification center would be more usable than utilizing existing components such as ephemeral banners or modals. Additionally, Prototype C emerged as the most successful of the Notification Center prototypes. Based on these findings, I was able to make a case for the value of adding a notification center to our platform, and succeeded in slotting this project in for that calendar year.

As the test results and analysis were for internal use only and the final designs have not been publicly implemented yet, I could not include them in my materials below. However, I would be happy to discuss my general approach, specific tests run, lessons learned, etc. upon request.

  • — category: UX/UI, research
  • — company: FinTech benefits platform
  • — year: 2021
  • — for: design direction & prioritization