User-Centered Research & Evaluation:
OnTime

Methodologies

As part of the course User-Centered Research & Evaluation, we were assigned to better the transportation experience of CMU students by narrowing into a focus/problem, performing various explorational then evaluative research methods, and creating a design artifact (whether it be a physical object, high-fidelity wireframes, etc.)

The problem area, provided to us by the Graduate Student Assembly, was: how can we improve transportation that students experience while attending CMU? After organizing previous design ideas through an effort vs. value axis, we arrived at bettering the experience of students who are habitually late to campus (with the intent of preventing their lateness). We then chose research methods through a methods selection table (seen at right, a helpful tool in planning how to answer research questions after defining a focus) from this population of students, which could then lead to design decisions later in our process.

Methods Selection Table (sample):

After we had defined our focus, we performed the following explorational research methods: contextual inquiry, affinity diagramming, diary study, think-aloud protocol, and speed-dating. I will only discuss the methods that I was involved in, although I am knowledgeable of the other methods performed on our team.


Explorational Methods

For our contextual inquiry, I interviewed a graduate student that regularly ran late to her 9:30 AM class. After creating an interview protocol, I performed the interview while taking the bus with her to her morning class. From the interview, I synthesized the following insights: students who live off campus have a hard time finding credible transportation apps, and being late can cause academic stress (especially when rushing to a test).

After the contextual inquiry, as a team, we created an affinity diagram to organize our insights from the two students we interviewed. First, we performed an interpretation session, in which we created two spreadsheets for each student & filled each with insights from the interview. Then, we grouped insights together to see a bigger picture of how students choose transportation methods when they are in a rush.

Affinity Diagram:

Affinity diagram

We then created a think-aloud protocol, with the purpose of finding how students use existing solutions (such as the apps Transit, Lyft) to get to campus when they are headed to class in the morning. After performing the protocol with a different graduate student, I created a usability aspect report (UAR) to summarize the critical incidents that occurred over the study. From this UAR, we found that the current solutions of Transit and Lyft are not adequate in allowing students to plan ahead (these solutions don't promote time management), and that Transit does not provide the ability to customize which busses are seen on the landing screen (which can be important when you are in a rush).

User scrolling through Transit landing screen, and attempting to plan ahead in Transit, accompanied with quote from Usability Aspect Report.

After performing explorational research methods, we created design artifacts that reflected our learnings from previous research. Because we were making instead of observing, we employed evaluative research methods that tested our understanding of the problem. These methods include: speed-dating, "walking the wall," desirability study, 5-second test, and a think aloud protocol. In the midst of these methods, we modified our design artifact to reflect our findings.

We started with many components to our design artifact, but narrowed down to these two over the course of these research methods: a dashboard, and a set of customization screens. The dashboard would give the user a concise recommendation for transit, as well as help with time management, and the customization screens would allow users to personalize the recommendations our solution provided.


Evaluative Research Methods:

In our desirability study, we wanted to understand the emotional responses that our customization section would evoke. Because our prototype contained a survey, we wanted to understand if it had too many questions, too little, or something else. After performing this test on 5 individuals, we found that the design method wasn't appropriate for our stage of research. We recieved helpful design feedback, but users were not emotionally affected by our artifact. The helpful feedback included clarifying confusing questions and eliminating redundancies.

Pre-Desirability Study

Post-Desirability Study

After our desirability study, we performed a 5-second test, as well as a think-aloud (in which we iterated on our design artifact in between the tests).

For the 5-second test, we tested the dashboard screen, which told the user which transportation method to take, and how much time they had left to leave. After performing 10 tests, we received valuable, specific feedback: our dashboard confused users about when they had to leave (there was no "time you need to leave," there was only a countdown). We adjusted our screens and moved on to our next method.

Pre vs. Post 5 Second Test

In our final method, we created a short think-aloud protocol to find if users understood how our customization screens worked and how their answers to the survey questions affected the dashboard. With an interactive prototype, we tested 2 users and created a UAR. From the UAR, we found that navigation through these screens was confusing. After making changes according to this method, we were ready to submit!


Solution

In addition to our design artifact, we also participated in a poster presentation with our classmates and clients, in which we gave an elevator pitch as well as fielded questions about our work.

Write-Up:

In Conclusion:

In this project, we selected and employed design methods that we had no prior experience with. We all learned by doing, and failed many times. After our failures, we learned from our mistakes, solidified our understanding of the problem, and more generally, how to best use a design method to gain insights about users in a problem stage or context. I also learned how to be a teammate: how to communicate, how to give meaningful, constructive criticism, and how to take it (criticism). We created something we & our professors were proud of.