UCSF Delirium Reduction Initiative

TL;DR

My team and I engaged in iterative design and usability studies to create an online dashboard to evaluate the performance of the UCSF Delirium Reduction Initiative.

Background

My UC Berkeley team worked with client University of California, San Francisco (UCSF) to build an online dashboard to evaluate the success of their Delirium Reduction Initiative. ​​​The​ ​goal​ ​of the​ ​​​online​ ​dashboard​ ​is​ ​to​ ​help​ ​the​ ​hospital’s​ ​directors​ ​evaluate​ ​the​ ​performance​ ​of​ ​the program​ ​and​ ​use​ ​the​ ​key​ ​metrics​ ​to​ ​decide​ ​which​ ​locations​ ​require​ ​funding​ ​to​ ​improve​ ​the overall program’s​ ​results​ ​and​ ​patients’​ ​well-beings.

Supplementary materials: Our online dashboard won “Best UI Design”, awarded to us by both the UCSF directors and our peers. Please feel free to take a look at our UCSF stakeholder interview guide, personas and task analysis, usability study protocol, usability study follow-up survey, and our final presentation to the UCSF stakeholders.

Problem Statement

Delirium​ ​is​ ​a​ ​serious​ ​disturbance​ ​in​ ​mental​ ​abilities​ ​that​ ​results​ ​in​ ​confused​ ​thinking​ ​and​ ​reduced awareness​ ​of​ ​your​ ​environment.​​ ​According​ ​to​ ​UCSF,​ ​25%​ ​of​ ​adult​ ​care​ ​patients​ ​develop delirium​ ​during​ ​their​ ​stay​ ​at​ ​UCSF.​ ​To combat this, ​UCSF​ ​implemented​ ​the​ ​Delirium​ ​Reduction Initiative​,​ ​a​ ​program​ ​to​ ​reduce​ ​the​ ​incidences​ ​of​ ​delirium​ ​at​ ​UCSF.​ ​​​UCSF​ ​wanted ​to understand​ ​the​ ​overall​ ​and​ ​per​ ​hospital​ ​location​ ​performance​ ​of​ ​the​ ​program, hence the partnership with UC Berkeley to build prototypes of an online dashboard to evaluate its program.

Research Question

Our main research question was:

• How might we design an online dashboard that meets and goes beyond UCSF directors’ and stakeholders’ goals of evaluating the performance of the Delirium Reduction Initiative?

Research Methodology

Timeline

We collaborated with UCSF over a span of four months.

• Discuss and plan the project: 1.5 week

• Low-fi prototyping: 0.5 weeks

• Interview stakeholders: 1 week

• Storyboarding: 0.5 weeks

• Personas and task analysis: 1.5 week

• 2nd iteration of prototyping (medium-fi prototyping): 2 weeks

• Recruitment: 1 week

• Usability testing: 2 weeks

• Analyze findings: 2 weeks

• Final design: 1 weeks

• Presentation: 1 week

Recruitment Process and Criteria

Our participants of the usability study were the UCSF directors invested in the outcomes of the UCSF Delirium Reduction Initiative. It took about a week to schedule the sessions.

Design Process

From the initial briefing of the project, we learned that the UCSF directors had two main goals:

• Gauging the effectiveness of the overall program

• Investigating which units are successful and which are problematic

We leveraged this knowledge in creating our low-fi prototypes, medium-fi prototypes, and final design.

1. Low-fi prototyping

In​ ​our​ ​low-fi prototype ​(Figure​ ​1),​ ​we​ ​designed​ ​the​ ​view​ ​by​ ​role:​ ​director,​ ​program​ ​manager,​ ​and nursing​ ​lead.​ ​When​ ​a​ ​user​ ​clicks​ ​on​ ​a​ ​view,​ ​information​ ​relevant​ ​to​ ​that​ ​role​ ​would​ ​be highlighted.​ ​We​ ​also​ ​had​ ​graphs​ ​that​ ​showed​ ​the​ ​decrease​ ​in​ ​delirium rate​ ​overtime​ ​and​ ​delirium​ ​rate​ ​for​ ​each​ ​location.​

2. Interview stakeholders

We interviewed our stakeholders to understand what was their goal and use-case for the online dashboard, using this interview guide. We asked open-ended questions to allow our stakeholder full freedom to explain their goals and needs for the online dashboard.

3. Storyboarding

From our conversations with our stakeholders, we learned that​ ​we​ ​should​ ​provide​ ​the​ ​view​ ​based​ ​on​ ​location​ ​instead​ ​of​ ​by​ ​role,​ ​hence​ ​the​ evolution into​program and​ ​location​ ​views.​ ​We also learned what metrics they are interested in seeing in the dashboard. In​ ​our​ ​storyboard​ ​(Figures​ ​2​ ​and​ ​3),​ ​we​ ​called out​ ​two​ ​key​ ​tasks: measuring​ ​the​ ​effectiveness​ ​of​ ​the​ ​overall​ ​program,​ ​and​ ​investigating​ ​which​ ​units​ ​are​ ​successful and​ ​which​ ​are​ ​problematic.​ ​In​ ​Figure​ ​2,​ ​the​ ​program​ ​view​ ​displays​ ​trends​ ​in​ ​delirium duration,​ ​intensity,​ ​patient​ ​outcomes,​ ​and​ ​compliance​ ​rate,​ ​allowing​ ​the​ ​user​ ​to​ ​gauge the​ ​overall​ ​performance​ ​of​ ​the​ ​program.​ ​In​ ​Figure​ ​3,​ ​​​the​ ​location​ ​view​ ​displays​ ​the​ ​same metrics​ ​as​ ​the​ ​program​ ​view​ ​by​ ​location,​ ​allowing​ ​the​ ​user​ ​to​ ​gauge​ ​which​ ​locations​ ​are successful​ ​or​ ​problematic.​ ​

4. Personas and task analysis

Our team created 3 personas (nursing lead, program manager, and director) to showcase the goals and motivations of our stakeholders, and a breakdown of the tasks they hoped to accomplish through the online dashboard. For brevity, I’ve only included 1 persona here, you can view the rest here.

June’s bio​:​ ​Dr.​ ​June​ ​Chan​ ​is​ ​senior​ ​director​ ​of​ ​nursing​ ​practice​ ​at​ ​UCSF​ ​Hospital.​ ​She received​ ​her​ ​BA​ ​in​ ​Applied​ ​Math​ ​in​ ​Biology​ ​from​ ​Harvard​ ​College​ ​and​ ​her​ ​doctoral degree​ ​from​ ​Harvard​ ​Medical​ ​School.​ ​At​ ​her​ ​age​ ​of​ ​45,​ ​she​ ​has​ ​already​ ​received​ ​many accolades​ ​for​ ​her​ ​groundbreaking​ ​research​ ​in​ ​how​ ​​external​ ​factors​ ​such​ ​as​ ​diet​ ​and exercise,​ ​and​ ​internal​ ​factors​ ​such​ ​as​ ​hormones​ ​and​ ​genetics,​ ​contribute​ ​to​ ​prostate cancer. As​ ​the​ ​director​ ​of​ ​nursing​ ​practice,​ ​she​ ​is​ ​in​ ​charge​ ​of​ ​the​ ​nursing​ ​units,​ ​as​ ​well​ ​as making​ ​sure​ ​all​ ​operations​ ​go​ ​smoothly.​ ​She​ ​also​ ​cares​ ​about​ ​the​ ​overall​ ​effectiveness of​ ​programs​ ​launched​ ​at​ ​UCSF,​ ​such​ ​as​ ​the​ ​new​ ​Delirium​ ​Reduction​ ​Initiative.​ ​She​ ​is also​ ​a​ ​researcher,​ ​professor​ ​at​ ​UCSF,​ ​wife,​ ​and​ ​mother​ ​to​ ​three​ ​young​ ​children.​ ​She’s not​ ​particularly​ ​tech​ ​savvy​ ​and​ ​a​ ​busy​ ​woman,​ ​so​ ​she​ ​needs​ ​an​ ​easy​ ​interface​ ​to explore​ ​the​ ​data​ ​pertaining​ ​to​ ​the​ ​Delirium​ ​Reduction​ ​Initiative.​ ​With​ ​an​ ​accessible interface,​ ​she​ ​can​ ​quickly​ ​decide​ ​the​ ​overall​ ​effectiveness​ ​of​ ​the​ ​program​ ​and​ ​whether or​ ​not​ ​to​ ​support​ ​the​ ​program​ ​moving​ ​into​ ​the​ ​next​ ​quarter.

June’s goals:​ ​​June’s​ ​goal​ ​is​ ​to​ ​easily​ ​see​ ​a​ ​high-level​ ​view​ ​of​ ​the​ ​Delirium​ ​Reduction Initiative’s​ ​performance.​ ​This​ ​will​ ​also​ ​help​ ​her​ ​to​ ​understand​ ​the​ ​financial​ ​performance of​ ​the​ ​program​ ​to​ ​make​ ​funding​ ​decisions.​ ​As​ ​for​ ​her​ ​personal​ ​aspirations,​ ​she​ ​wants to​ ​not​ ​only​ ​see​ ​patients’​ ​successes​ ​at​ ​UCSF,​ ​but​ ​also​ ​demonstrate​ ​her​ ​leadership​ ​and decision​ ​making​ ​skills​ ​to​ ​the​ ​UCSF​ ​board​ ​of​ ​directors.

Tasks for June:

• Correlate results to show program’s overall effectiveness (Must-have)

• Visualize overall program performance overtime (Must-have)

• Funding decisions for future investment of the program (Nice-to-have)

5. 2nd iteration of prototyping (medium-fi prototyping)

During​ ​our​ ​client​ ​meeting,​ ​the​ ​client​ ​expressed​ ​a​ ​desire​ ​to​ ​see​ ​information​ ​regarding​ ​the performance​ ​of​ ​the​ ​UCSF​ ​Reductive​ ​Initiative​ ​overall​ ​and​ ​per​ ​location/unit.​ ​Therefore,​ ​we decided​ ​to​ ​provide​ ​two​ ​views:​ ​one​ ​view​ ​that​ ​describes​ ​key​ ​overall​ ​metrics​ ​​​and​ ​graphs,​ ​and another​ ​view​ ​that​ ​describes​ ​specific​ ​metrics​ ​and​ ​graphs​ ​to​ ​each​ ​location/unit. For​ ​the​ ​program​ ​view,​ we wanted to display​ ​a program outcomes graph (Figure 4) and a patient​ ​outcomes​ ​graph (Figure 5).

For the location/unit view, our goal was to display which units were successful, and which were problematic and needed more resources. Therefore we showed key metrics, such as delirium intensity, days, and intensity specific to each location (Figure 6), and also how many patients were delirious per location (Figure 7).

6. Usability testing

We conducted usability testing of our prototype with 3 stakeholders, using this protocol guide. I facilitated the study, while my teammates took notes and observed. We led our participants through 3 scenarios:

• Investigate which units are successful and which are problematic

• Measure the effectiveness of the overall Delirium Reduction Initiative

• Decide how to best allocate funds for the Delirium Reduction Initiative

During each scenario, I asked our participants a series of questions about the tasks. We also measured their performance (how many correctly answered questions). At the end of our study, we asked our participants to fill out a survey about the ease of use and clarity of our prototype. To conclude our study, we thanked our participants for their time.

Our main learning from the usability study was that users wanted the​ ​program’s​ ​trend​ ​in​ ​delirium​ ​duration​ ​and​ ​length​ ​of stay​ emphasized ​because​ ​UCSF​’s goal is for ​those metrics​ to decrease.​

Findings

Our main finding was that our users wanted the dashboard to focus on delirium duration and length of stay for both the program and location views, as those are the key metrics in determining the success of the program. Our client also wanted to see trends in compliance rate for both the program and location views, and clear captions describing each graph. We took this feedback and created our final design.

Final Design

We built our online dashboard using Tableau, hosting it on GitHub. In response to the feedback, for the program view, we focused one graph on average length of stay and delirium duration (Figure 8), and another graph on program compliance (Figure 9). We added trend lines and detailed captions to both graphs.

For the location/unit view, we added baselines to the delirium metrics graph (Figure 10), aiming to make it clearer if a unit is above or below target. We also added detailed captions.

Impact

We​ ​hope​ ​that our​ ​the​ ​final designs ​allowed​ ​the​ ​user​ ​to​ ​see​ ​the​ ​overall​ ​and​ ​location performance​ ​of​ ​the​ ​program,​ ​and​ ​aid​ ​in​ ​funding​ ​decisions.