This article is re-posted from User Centric’s website.
Chicago-based usability firm User Centric (now GfK’s User Experience group) conducted a series of three usability studies with iPhone customers who purchased their iPhones on the first day of its US release. The first study began within days of the release and examined overall ease-of-use factors and general usability issues among motivated early adopters who had limited experience with the iPhone. The second study focused on the iPhone's touch keyboard in comparison with conventional QWERTY and numeric phone keyboards. The third study compared the texting experiences of iPhone owners and non-owners across devices.
Study #1: July 2007
Download User Centric’s Mini-Report (Study 1)
The goal of this study was to identify overall ease-of-use factors and general usability issues among motivated early adopters who had limited experience with the iPhone.
Six days after the iPhone was launched, User Centric (now GfK’s User Experience group) tested the usability of the iPhone during 60-minute usability sessions with these iPhone customers. The participants, who had the same short-term exposure to their new iPhones, were asked to carry out the same set of tasks using their new iPhones. We also asked participants to compare text entry on their iPhone with their prior mobile phones (which used multi-tap text entry because their old phones did not have QWERTY keypads).
Test participants were recruited from iPhone customers who purchased their devices on June 29, 2007 in the Chicago metro area. Participants were required to bring their personal iPhones to the study as proof of ownership. None had used an iPhone prior to June 29. All participants were compensated for their time and none were employed, related to, or personally known to User Centric prior to this study.
Based on test data from these short-term iPhone users, we identified the following strengths and weaknesses in the iPhone's user experience:
Major strengths of the iPhone's user interface
Participants found the Visual Voicemail feature to be intuitive and useful
Participants compared text entry using their iPhone and their previous phones (which used multi-tap for text entry). Overall, participants found that text entry was much easier on the touchscreen soft-keyboard of the iPhone compared to standard multi-tap text entry.
Most participants used the landscape (horizontal) view while searching for a website (The New York Times.) The horizontal soft keyboard was definitely preferred over the vertical keyboard orientation.
Selecting and playing a song using both the vertical and horizontal views was easy for participants. Participants also commented that the 'Cover Flow' navigation feature was a very engaging feature.
Making an outgoing call with the iPhone was easy for all participants.
Participants also found that receiving a phone call on an iPhone while listening to music was seamless. They found the interruption of the music and transition to a call to be very smooth.
Saving a number as a contact and recalling that contact for a phone call was straightforward.
Answering a phone call while in SMS mode was also straightforward. Participants found it helpful to be taken immediately to their message after the call had concluded.
Most participants found it easy to take a picture and email it from the iPhone.
Primary difficulties with the iPhone interface
Participants uniformly found text entry SMS and email to be difficult. They were frustrated by the forced use the vertical keyboard and the lack of visibility for editing the middle of a word or sentence.
Many participants found Google Maps difficult to use on the iPhone. They experienced issues with the fine-motor control required to pan accurately in different dimensions in Google Maps and predictably zoom in and out. It was unclear to participants how much they needed to adjust the size of their "pinch" gesture to zoom in and out with the control that they wanted.
Participants were often frustrated with their Web browsing experience and hoped that this would improve dramatically with an upgrade in network capability. The lack of Flash and Java capabilities during Web browsing was considered a detriment to basic Web use.
Finally, participants were surprised (and somewhat annoyed) to discover that horizontal text entry was available only in in the Safari browser.
As part of their sessions, participants were also asked to rate their first time usage of the iPhone for key tasks. This data will be used as just one of our baseline metrics for iPhone longitudinal studies.
Study #2: August 2007
Download User Centric’s Mini-Report (Study 2)
In the earlier baseline study, we identified texting as potentially problematic for new iPhone customers. The goal of the current study was to determine how easy it was for conventional mobile phone users to text using the iPhone.
A total of 20 participants were brought in for one-on-one usability sessions with a moderator. All sent text messages at least 15 times per week. Ten of the participants owned a phone with a QWERTY keypad, and ten of the participants owned a phone with a numeric keypad. Those who owned a numeric keypad used the "multitap" method of entering text messages rather than predictive text. To multi-tap, a user must press a particular key on the numeric keypad multiple times to get the desired character to appear.
During each session, participants were required to use their own phones. In addition, they were provided with an iPhone for use during the study. None of the participants were iPhone owners, and all participants were compensated for their time.
Every participant entered six messages using their own phone and six messages on the iPhone. In their sessions, participants were asked to copy 12 standard messages created for this study. Each message was between 104-106 characters in length (including spaces). Six of the messages each contained 8-10 instances of proper capitalization and punctuation, while the remaining six messages contained no capitalization or punctuation but had some abbreviations. Message configuration and phone order were counterbalanced across participants.
We were aware that participants' prior familiarity with their own phones meant that there would likely be a learning curve associated with text messaging on the iPhone. (None of the participants had used an iPhone prior to the study). Although participants were given one minute to familiarize themselves with the iPhone's touch keyboard, their texting abilities on the iPhone were still at the novice level. Throughout the study, we did notice limited improvements in keyboard comfort as users progressed through the tasks on the iPhone. Overall, the findings in the study can be taken as a good representation of what iPhone text messaging is like for a customer who has just bought an iPhone and is using it for the first time.
"It's important to consider the changes a person has to make when they switch to the iPhone," said Gavin Lew, (EVP, User Experience, GfK). "It should be easy for people to do common tasks, such as text messaging, using the iPhone's less traditional touch interface."
Texting on iPhone Took Twice as Long as Texting on QWERTY Phones
In general, participants took longer to enter text messages on the iPhone than on their own phone. Despite the keyboard similarities, QWERTY phone users took nearly twice as long to enter comparable messages on the iPhone compared to their own phone. On the other hand, multi-tappers did not experience a significant difference in the time it took them to type messages on the iPhone. (Multi-tap text entry usually takes about 2 times as long as QWERTY text entry). Participants also made more typing errors on the iPhone. This phenomenon was expected since users had much more experience with their own phones.
Detailed Observations of User Text Entry on the iPhone:
Most participants felt that their fingertips were too large for the iPhone's touch keyboard.
Most QWERTY phone users initially used the iPhone by holding it with both hands and typing with their two thumbs. However, by the end of the session, most had decided that it was easier for them to use one index finger to type.
Over half of the participants stated that they would have preferred the feel of an actual key to the iPhone's touch keypad.
Most participants noticed that there was no tactile feedback on the iPhone keypad.
Some mentioned that the feel of the key on conventional phones helps them locate the desired key without having to focus on the actual keypad.
General Interface Issues
Participants expressed a great deal of frustration with the sensitivity of the iPhone touch keypad.
Participants made an average of 11 errors per message on the iPhone compared to an average of 3 errors per text message on their own phone. Although the error rate was alleviated somewhat by the iPhone's self-correction feature, participants were still frustrated.
In particular, participants struggled when they were trying to type using the Q & W keys or the O & P keys on the iPhone.
5 out of 20 participants asked if the iPhone came with a stylus. They indicated that they could be more accurate with the stylus rather than their fingers due to the sensitivity of the screen.
One female participant tried to interact with the iPhone keypad using her fingernail and was unsuccessful.
The space bar, return, and backspace keys presented issues for many participants because these keys were spaced so closely to each other.
No one discovered the drag and lift feature of the keyboard, which reduces errors.
Many participants said they could not see themselves attempting text entry on the iPhone in distracting conditions.
Specifically, participants did not think they could text message on the iPhone safely while driving.
Predictive & Corrective Text Issues
Only a few participants discovered and correctly learned to use the predictive and/or corrective text features on the iPhone. QWERTY phone users in particular had a tendency to backspace when they were correcting mistakes.
Participants did not understand how the predictive / corrective text bubbles worked.
6 out of 20 participants tried to touch the bubble to get the word in the text bubble to appear.
Three participants tried hitting the backspace key because they associated the 'x' on it with the 'x' in the bubble.
It was especially frustrating for participants when they attempted to place the cursor in the middle of a word.
None of the participants discovered the magnifying glass feature while text messaging.
·During a follow-up task that involved correcting a note in the iPhone's Notes application, 6 out of 20 did discover the magnifying glass feature. However, not all participants realized that the feature helps place the cursor in addition to enlarging the text.
Real World Implications
Based on our study's findings, it appears that QWERTY phone users are likely to suffer some initial decrease in efficiency when switching to the iPhone touch keyboard. However, multi-tap texters may see an eventual increase in text entry efficiency when switching to the iPhone.
Our study indicates that both QWERTY and multi-tap users are likely to have some level of initial frustration with the iPhone's touch keyboard and corrective text features. Although our analysis suggests that both types will eventually adapt to the iPhone's features, the learning curve for texting on iPhones will be steeper for QWERTY phone users than multi-tappers.
Study #3: November 2007
Download User Centric’s Mini-Report (Study 3)
Previously, User Centric found that overall design and usability of the iPhone was good, but the iPhone's touch keyboard was a weak point for many users. The third study examined specific interactions with the iPhone touch keyboard and compares the texting experiences of iPhone owners and non-owners across devices.
Our study involved data from 60 participants who were asked to enter specific text messages and complete several mobile device tasks. Twenty of these participants were iPhone owners who owned their phones for at least one month. Twenty more participants were owners of traditional hard-key QWERTY phones and another twenty were owners of numeric phones who used the "multi-tap" method of text entry.
Participants were brought in for 75-minute usability sessions. Each participant entered six fixed-length text messages on their own phone. Non-iPhone owners also did six messages each on the iPhone and a phone of the 'opposite' type. The opposite phone for numeric phone owners was a Blackberry and for hard-key QWERTY phone owners it was a numeric Samsung E300 phone. Some participants did additional tasks, including a contact search and add contacts, as time allowed.
Texting Performance of Participants Compared on Multiple Devices
This study used a mixed factorial design to compare the performance of the different types of phone owners while creating text messages on different types of phones (iPhone, hard-key QWERTY, and numeric phones). We defined performance in terms of time to complete tasks and number of errors made per text message.
iPhone owners entered six text messages on their own phone. They also typed two pangrams - a sentence that includes every letter in the English language at least once - and one corpus - a set of characters that represents the exact letter frequencies of the English language. These tasks were included to ensure that participants experienced the various phone keyboards in a thorough manner. iPhone owners also completed tasks involving text correction, contacts, and visual voicemail
Non-iPhone owners entered a total of 18 text messages - six each on their own phone (hard-key QWERTY or numeric phone), the iPhone, and the 'opposite' phone (numeric test phone for QWERTY phone owners, hard-key QWERTY test phone for numeric owners). These participants also entered two pangrams and one corpus on their own phone and completed the contact list tasks if time was remaining.
The order of phones and text messages were counterbalanced across participants to prevent ordering effects. The text messages contained 104-106 characters, including spaces. Half of the messages contained 8-9 instances of capitalization and punctuation while the other half had none. Messages were never seen more than once by a participant. The pangram tasks contained 66-67 characters and the corpus consisted of 112 characters.
At the end of the session, all participants rated each phone and ranked them in order of ease of use for text messaging.
iPhone and Hard-Key QWERTY Texting Was Equally Rapid, but iPhone Owners Made More Errors
When compared to hard-key QWERTY phone owners using their personal phones, iPhone owners' rate of text entry on the iPhone was equally rapid. However, iPhone owners made more errors during text entry and also left significantly more errors in the completed messages.
While iPhone owners made an average of 5.6 errors/message on their own phone, hard-key QWERTY owners made an average of 2.1 errors/message on their own phone, p < .01. iPhone owners also left an average of 2.6 errors/completed message created on the iPhone compared to an average of 0.8 errors/completed message left by hard-key QWERTY phone owners on their own phoneInterestingly, when comparing the performance of iPhone owners and novices (non-iPhone owners), there was no significant difference between the number of errors made, p = .21. iPhone owners were faster than non-iPhone owners, of course."Despite the correction features available on the iPhone, this data suggests that people who have owned it for a month are still making about the same number of errors as the day they got it," says Gavin Lew, (EVP, User Experience, GfK).Furthermore, when iPhone owners were asked to perform a text correction task during their sessions, 21% of iPhone owners were not aware of the magnifying glass correction feature although they had owned their iPhone for one month. Participants who did know about the feature clearly loved it, and participants who were new to it indicated that it would be useful in the future.
Numeric Phone Owners Texted More Accurately on Unfamiliar QWERTY Phone than Unfamiliar iPhone
Participants who had previously not used either a hard-key QWERTY phone or an iPhone were significantly faster at entering text messages on the hard-key QWERTY test phone than on the iPhone. These participants also made significantly fewer errors on the hard-key QWERTY than on the iPhone.
Numeric phone owners made an average of 5.4 errors/message on the iPhone, 1.2 errors/message on the QWERTY test phone, and 1.4 errors/message on their own phone.
"Not only was their performance better," says Jen Allen (User Experience Specialist, User Centric) "their rankings and ratings of the phones indicated that they preferred a hard-key QWERTY phone for texting."
Participants rated the hard-key QWERTY phone highest out of all three phones for ease-of- text messaging. The hard-key QWERTY phone was also most frequently ranked first out of the three phones by the numeric and QWERTY users. Overall, the hard-key QWERTY phone was ranked first in text messaging by 85% of users. iPhone was ranked second by 60% of these users. None of the hard-key QWERTY phone owners ranked the iPhone first for text messaging and only three numeric phone owners ranked the iPhone first.
Detailed Analysis Points to Common User Errors on iPhone Keyboard
As part of a detailed analysis of text entry patterns on the iPhone, we reviewed videos of each participant session to classify and count errors that were made by participants during the iPhone text entry tasks. When a single letter was omitted, incorrectly inserted, or substituted, this was counted as an error. If the iPhone corrective text feature made an improper correction, this was still counted as a single error even if multiple letters were changed.
A matrix was constructed using data from about 34 participants using the iPhone, both owners and non-owners. The matrix allowed us to compare the letters that participants intended to enter (based on the task) with the actual letters entered. Afterwards, we identified hits, misses, false alarms, and correct rejections for each letter on the iPhone keyboard.
In general, hit rates for all keys on the iPhone keyboard were consistently 90% or higher. Hit rate refers to the percentage of time that a key was correctly pressed when it was intended. The 'W' key had the lowest hit rate, while the 'Q' key had the highest hit rate. The average hit rate was about 95%. To generalize, the keys on the outside of the keyboard, such as Q, A,, Z, and P, L, and M, had high hit rates.
However, the false alarm rates indicated that participants were repeatedly pressing certain keys when they intended instead to press other adjacent keys. For example, a false alarm for a given letter is said to have occurred if a participant meant to press 'W' and instead pressed 'Q'. This would count as a miss for 'W' and a false alarm for 'Q'. False alarms are relevant because they increase the time used to enter and correct a text message.
Several iPhone keys had high false alarm rates: Q (66%), P (27%), J (22%), X (21%), and Z (15%). In contrast, the median false alarm rate across the iPhone entire keyboard was 5.48%.
iPhone keys with the highest false alarm rates were those in close proximity to the five most frequently used letters in the English language - E, T, A, O, and I. In addition to the high false alarm letters listed above, other false alarm letters included W (10%), R (6.5%), Y (8.7%), and S (6.0%), which are also adjacent to high-frequency letters. B (8.2%) also had a high false alarm rate, potentially because of its location near the letter N (which is the sixth most frequent letter).
Detailed Analysis of Errors on Hard-Key QWERTY Phone Keyboard
We conducted similar analysis with user errors on the hard-key QWERTY phone keyboard. A matrix was constructed using data from 15 hard-key QWERTY phone owners from tasks using their own phones. The main purpose of constructing this matrix was to be able to compare the iPhone keyboard to more established QWERTY phone keyboards.
On the hard-key QWERTY keyboard, the hit rates for all keys were above 97%, except for V (96%). Additionally, the false alarm rates for keys on this keyboard were below 3%, with the exception of Q (8%). Performance on this keyboard with respect to errors was much better than on the iPhone keyboard. The letters with higher false alarm rates were similar on both keyboards, involving many of the 5 least frequently used letters in the English language, such as Q, Z, V, and B. Also, the Q and P keys were problematic for users of both keyboards, suggesting that the issue for these keys arises from their location near the top edges of the keyboards.
Participants made different types of errors on the iPhone and the hard-key QWERTY phones. The majority of errors made on the iPhone involved substituting a nearby letter for the intended letter. However, on the QWERTY phone, participants made more insertion and omission errors than substitution errors. Also, many of the substitution errors that were made on the QWERTY keyboards involved swapping the order of the correct letters in the words, such as typing "stomr" instead of "storm".
iPhone May Not be Suitable for Heavy Text Use
Compared to hard-key QWERTY devices, the iPhone may fall short for consumers who use on their mobile device heavily for email and text messaging. The iPhone was clearly associated with higher text entry error rates than a hard-key QWERTY phone. The finding that iPhone owners made more texting errors on iPhones than their hard-key QWERTY counterparts (on their own QWERTY phones) suggests that the iPhone may have a higher fundamental error rate. Specifically, the high rate of false alarms for iPhone keys adjacent to high frequency letters is troubling. The iPhone's predictive and corrective text features do alleviate some of the errors users make while texting, but it does not catch them all.
"The iPhone is a great switch from a numeric phone. But if you're switching from a hard-key QWERTY phone, try the iPhone in the store first," recommended Lew.
User Centric received a number of inquiries with questions about this research. Please find the FAQs below posted in an effort to provide more information to further clarify the research findings. We hope these findings are interesting and serve to stimulate discussions to make mobile devices more useful and engaging for consumers.
1) Was this research funded by another party?
Answer: No. This study was entirely funded by User Centric with zero client involvement / sponsorship.
2) Why would you do this research?
Answer: The goal was to publish findings for public consumption in an effort to further mobile phone research and answer interesting user experience questions. Mobile technology is a core area of specialty for us. In the last few years, we completed well over one hundred projects in the mobile space. Our user research on small screen devices have involved over 2,500 individual, one-on-one participant sessions (60 to 120 min sessions -- not online surveys) across five continents and over 20 countries. As such, we are always interested in new technologies and wanted to learn more.
3) Why was the sample size (N=20) so low?
Answer: As with any research study, the question of sample size and power must be addressed (i.e., Is the sample large enough to identify a significance, if it does indeed exist). The simple answer is: Yes. The sample size was sufficiently large and all results presented were statistically significant.
The long-winded answer is: Yes. While the sample may seem low by some large scale marketing standards, I can say with confidence that for hypothesis testing, the sample is certainly sufficient. Again, the fact that the primary results presented were statistically significant should tell those who conduct studies that the findings are legitimate.
Our methods employed a 2x2 mixed factorial design. Type of phone owned by the participant was varied between subjects. However, participants typed text messages on both their own phone and the iPhone during the test session. Message length, phone order, and message order were both controlled.
With hindsight, we should have included the stats behind the results, but we did not want to over complicate the press release.
4) What should the typical mobile phone user expect as a transition period?
Answer: For the general mobile phone user, any new device will have a transition period. However, in terms of the text input system, what this study found was that there was NO difference in text messaging efficiency between a mobile phone (non-QWERTY) compared to the iPhone's touch keyboard. This is compelling because it suggests that users will have the same texting efficiency on the iPhone as they do today. Moreover, the first iPhone study we did on iPhone owners one week after they switched found that their text efficiency was better than the mobile phone they used just one week earlier.
That said, the same finding was not found for QWERTY device owners.
5) What happened with those who use QWERTY devices? Please tell me more about these users.
Answer: We specifically recruited participants who owned full QWERTY mobile devices. Thus, the findings that QWERTY users were 2x more efficient in texting should not be that surprising. A physical keyboard, which has the affordances of tactile feedback, should TRUMP a glossy virtual keyboard. I do not think that even Steve Jobs would argue that this study's finding is shocking.
6) Participants text at least 15 messages per week. That seems low. Can you elaborate?
Answer: To recruit participants, we created a recruiting screener and the 15 message minimum was a criterion in the screener. The mean number of text messages sent per week was 165.8 for all participants. For QWERTY device and numeric keypad phone users, means were 225.1 and 106.4 text messages per week, respectively. So, these participants did text a great deal (over 400 per month).
So why did we do this research – for free? Because we have a passion for user research and devices and the iPhone was the first mainstream mobile device to incorporate multi-touch technology. We're an independent user research company that likes gadgets and likes gadgets to be usable.
We also happen to be staffed by email and texting addicts. So it was logical for us to conduct some independent research to examine overall user experience of this technology.