Poster: The Diary Sandwich: Rich Mobile Data on a Budget
A Sample Submission from Vito Loconte
Poster: The Diary Sandwich: Rich Mobile Data on a Budget
Vito Loconte |
Short Description
The diary sandwich is a methodology that consists of a longitudinal diary study sandwiched between two usability tests. We used this methodology in two different first-time use studies to uncover initial usability issues as well as to gather feedback on behaviors and issues after the initial use and setup. Participants submitted diary entries using the free Google Voice service, which also transcribed the diary entries, saving the researchers time and effort when compared to other voicemail diary studies. We also used additional software (GoToMeeting, Morae, and Linoit) to collaborate with stakeholders during and after the usability test sessions. Long Description Diary studies and mixed methods studies are becoming the new norm for collecting rich data about complex user experiences, especially in the mobile space. As computing in general falls off the desktop into mobile devices, it’s become increasingly critical that we understand experiences people have while on the go. This presents challenges for UX professionals: it’s harder to do direct observation than, say, a workplace or site visit. Mobile diary studies and mixed methods studies offer an alternative approach. In this poster, I will show how to make data collection much faster and cheaper, while also increasing the quality of data captured. At <Company>, we have conducted two studies in the past year that combine a digital diary study with multiple usability test sessions. In both cases, our goal was to observe the first-time use of somewhat complex systems: a remote support and IT monitoring system that takes 24 to 48 hours to collect information, and a cloud file storage and management system that also had a mobile app. Our goal for these first-time use studies was not only to uncover initial usability issues with the setup of the software, but also to observe and collect user behavior and feedback on their experience for the first few days that they interacted with the system. This will be a practical poster session that will give concrete how-to information and tips. I will discuss the methodology that we used and how we used some newer technology to iterate on previous digital diary and voicemail diary studies. I will also focus on how we used software to collaborate with stakeholders during and after the usability test sessions. We developed a methodology dubbed “The Diary Sandwich”. This method consists of a longitudinal diary study sandwiched between two usability tests. One of the reasons we chose this method was so we could observe some existing setup issues that we knew were happening but didn’t know why. Also, since both pieces of software took a number of days to collect and store information, we wanted to learn how users interacted with the software in that time and what issues they had after the initial setup. And finally, we brought them back for a second usability test to talk about the features and functions they used during their initial exploration and also to give them some more advanced tasks to perform now that they were more familiar with the products. In the second study, we iterated on our methodology and incorporated a way for participants to submit diary entries via voicemail. We also expanded on some past voicemail diary studies by incorporating some new technology, Google Voice. Since our second study had a mobile app that we wanted to collect data about, the addition of Google Voice was very important. This made it easy for participants to leave a voicemail that came directly to us, allowing them to submit diary entries on the go from their mobile devices. Google Voice was an important addition to this methodology also because it transcribed the voicemails, saving us valuable time and effort over previous voicemail diary studies. In addition to Google Voice (to record voicemail diary entries) and GoToMeeting (to conduct the remote sessions with the participants), we also used other software during the sessions to facilitate collaboration among the stakeholders:
We found that in the first study we conducted (where participants completed and submitted their journal entries through an online survey tool) that the majority of diary entries were brief, relatively infrequent, and weren’t very detailed. When we switched to Google Voice to collect the diary entries, we found that participants provided more robust and lengthy feedback, and also submitted a higher number of average entries. In both studies, we shared the diary entries with stakeholders before the follow-up sessions, so they could see the type of tasks that the participants performed and the issues they encountered. Goals Attendees will leave with some new ideas about how to conduct their own diary or mixed methods study to benefit their own research for complex user experiences and products. They will also leave with some new ideas about how to incorporate tools similar to the ones that we used (Google Voice, GoToMeeting, Morae, and Linoit) to run budget friendly in-person and/or remote usability sessions. Finally, they will leave with ideas on how to provide their own virtual observation room to better work with distributed teams and also to facilitate stakeholder participation and collaboration in their studies. Selected Reviewer Comments
|