Study Finds Usability Problems with Parental Controls

User Centric, Inc., a Chicago-based usability consultancy, finished a third and final study examining the user experience of Apple's iPhone. Previously, User Centric found that overall design and usability of the iPhone was good, but the iPhone's touch keyboard was a weak point for many users. The current study examines specific interactions with the iPhone touch keyboard and compares the texting experiences of iPhone owners and non-owners across devices.

Study Looked at Three Types of Phone Owners

Our study involved data from 60 participants who were asked to enter specific text messages and complete several mobile device tasks. Twenty of these participants were iPhone owners who owned their phones for at least one month. Twenty more participants were owners of traditional hard-key QWERTY phones and another twenty were owners of numeric phones who used the "multi-tap" method of text entry.

Participants were brought in for 75 minute one-on-one usability sessions with a moderator. Each participant entered six fixed-length text messages on their own phone. Non-iPhone owners also did six messages each on the iPhone and a phone of the 'opposite' type. The opposite phone for numeric phone owners was a Blackberry and for hard-key QWERTY phone owners it was a numeric Samsung E300 phone. Some participants did additional tasks, including a contact search and add contacts, as time allowed.

Texting Performance of Participants Compared on Multiple Devices

This study used a mixed factorial design to compare the performance of the different types of phone owners while creating text messages on different types of phones (iPhone, hard-key QWERTY, and numeric phones). We defined performance in terms of time to complete tasks and number of errors made per text message.

iPhone owners entered six text messages on their own phone. They also typed two pangrams - a sentence that includes every letter in the English language at least once - and one corpus - a set of characters that represents the exact letter frequencies of the English language. These tasks were included to ensure that participants experienced the various phone keyboards in a thorough manner. iPhone owners also completed tasks involving text correction, contacts, and visual voicemail

Non-iPhone owners entered a total of 18 text messages - six each on their own phone (hard-key QWERTY or numeric phone), the iPhone, and the 'opposite' phone (numeric test phone for QWERTY phone owners, hard-key QWERTY test phone for numeric owners). These participants also entered two pangrams and one corpus on their own phone and completed the contact list tasks if time was remaining.

The order of phones and text messages were counterbalanced across participants to prevent ordering effects. The text messages contained 104-106 characters, including spaces. Half of the messages contained 8-9 instances of capitalization and punctuation while the other half had none. Messages were never seen more than once by a participant. The pangram tasks contained 66-67 characters and the corpus consisted of 112 characters.

At the end of the session, all participants rated each phone and ranked them in order of ease of use for text messaging.

iPhone and Hard-Key QWERTY Texting Was Equally Rapid, but iPhone Owners Made More Errors

When compared to hard-key QWERTY phone owners using their personal phones, iPhone owners' rate of text entry on the iPhone was equally rapid. However, iPhone owners made more errors during text entry and also left significantly more errors in the completed messages.

While iPhone owners made an average of 5.6 errors/message on their own phone, hard-key QWERTY owners made an average of 2.1 errors/message on their own phone, p < .01. iPhone owners also left an average of 2.6 errors/completed message created on the iPhone compared to an average of 0.8 errors/completed message left by hard-key QWERTY phone owners on their own phone.

Interestingly, when comparing the performance of iPhone owners and novices (non-iPhone owners), there was no significant difference between the number of errors made, p = .21. iPhone owners were faster than non-iPhone owners, of course.

"Despite the correction features available on the iPhone, this data suggests that people who have owned it for a month are still making about the same number of errors as the day they got it," says Gavin Lew, Managing Director.

Furthermore, when iPhone owners were asked to perform a text correction task during their sessions, 21% of iPhone owners were not aware of the magnifying glass correction feature although they had owned their iPhone for one month. Participants who did know about the feature clearly loved it, and participants who were new to it indicated that it would be useful in the future.

Numeric Phone Owners Texted More Accurately on Unfamiliar QWERTY Phone than Unfamiliar iPhone

Participants who had previously not used either a hard-key QWERTY phone or an iPhone were significantly faster at entering text messages on the hard-key QWERTY test phone than on the iPhone. These participants also made significantly fewer errors on the hard-key QWERTY than on the iPhone.

Numeric phone owners made an average of 5.4 errors/message on the iPhone, 1.2 errors/message on the QWERTY test phone, and 1.4 errors/message on their own phone.

"Not only was their performance better," says Jen Allen, User Experience Specialist for User Centric, "their rankings and ratings of the phones indicated that they preferred a hard-key QWERTY phone for texting."

Participants rated the hard-key QWERTY phone highest out of all three phones for ease-of- text messaging. The hard-key QWERTY phone was also most frequently ranked first out of the three phones by the numeric and QWERTY users. Overall, the hard-key QWERTY phone was ranked first in text messaging by 85% of users. iPhone was ranked second by 60% of these users. None of the hard-key QWERTY phone owners ranked the iPhone first for text messaging and only three numeric phone owners ranked the iPhone first.

Detailed Analysis Points to Common User Errors on iPhone Keyboard

As part of a detailed analysis of text entry patterns on the iPhone, User Centric reviewed videos of each participant session to classify and count errors that were made by participants during the iPhone text entry tasks. When a single letter was omitted, incorrectly inserted, or substituted, this was counted as an error. If the iPhone corrective text feature made an improper correction, this was still counted as a single error even if multiple letters were changed.

A matrix was constructed using data from about 34 participants using the iPhone, both owners and non-owners. The matrix allowed us to compare the letters that participants intended to enter (based on the task) with the actual letters entered. Afterwards, we identified hits, misses, false alarms, and correct rejections for each letter on the iPhone keyboard.

In general, hit rates for all keys on the iPhone keyboard were consistently 90% or higher. Hit rate refers to the percentage of time that a key was correctly pressed when it was intended. The 'W' key had the lowest hit rate, while the 'Q' key had the highest hit rate. The average hit rate was about 95%. To generalize, the keys on the outside of the keyboard, such as Q, A,, Z, and P, L, and M, had high hit rates.

However, the false alarm rates indicated that participants were repeatedly pressing certain keys when they intended instead to press other adjacent keys. For example, a false alarm for a given letter is said to have occurred if a participant meant to press 'W' and instead pressed 'Q'. This would count as a miss for 'W' and a false alarm for 'Q'. False alarms are relevant because they increase the time used to enter and correct a text message.

Several iPhone keys had high false alarm rates: Q (66%), P (27%), J (22%), X (21%), and Z (15%). In contrast, the median false alarm rate across the iPhone entire keyboard was 5.48%.

iPhone keys with the highest false alarm rates were those in close proximity to the five most frequently used letters in the English language - E, T, A, O, and I. In addition to the high false alarm letters listed above, other false alarm letters included W (10%), R (6.5%), Y (8.7%), and S (6.0%), which are also adjacent to high-frequency letters. B (8.2%) also had a high false alarm rate, potentially because of its location near the letter N (which is the sixth most frequent letter).

Detailed Analysis of Errors on Hard-Key QWERTY Phone Keyboard

We conducted similar analysis with user errors on the hard-key QWERTY phone keyboard. A matrix was constructed using data from 15 hard-key QWERTY phone owners from tasks using their own phones. The main purpose of constructing this matrix was to be able to compare the iPhone keyboard to more established QWERTY phone keyboards.

On the hard-key QWERTY keyboard, the hit rates for all keys were above 97%, except for V (96%). Additionally, the false alarm rates for keys on this keyboard were below 3%, with the exception of Q (8%). Performance on this keyboard with respect to errors was much better than on the iPhone keyboard. The letters with higher false alarm rates were similar on both keyboards, involving many of the 5 least frequently used letters in the English language, such as Q, Z, V, and B. Also, the Q and P keys were problematic for users of both keyboards, suggesting that the issue for these keys arises from their location near the top edges of the keyboards.

Participants made different types of errors on the iPhone and the hard-key QWERTY phones. The majority of errors made on the iPhone involved substituting a nearby letter for the intended letter. However, on the QWERTY phone, participants made more insertion and omission errors than substitution errors. Also, many of the substitution errors that were made on the QWERTY keyboards involved swapping the order of the correct letters in the words, such as typing "stomr" instead of "storm".

iPhone May Not be Suitable for Heavy Text Use

Compared to hard-key QWERTY devices, the iPhone may fall short for consumers who use on their mobile device heavily for email and text messaging. The iPhone was clearly associated with higher text entry error rates than a hard-key QWERTY phone. The finding that iPhone owners made more texting errors on iPhones than their hard-key QWERTY counterparts (on their own QWERTY phones) suggests that the iPhone may have a higher fundamental error rate. Specifically, the high rate of false alarms for iPhone keys adjacent to high frequency letters is troubling. The iPhone's predictive and corrective text features do alleviate some of the errors users make while texting, but it does not catch them all.

"The iPhone is a great switch from a numeric phone. But if you're switching from a hard-key QWERTY phone, try the iPhone in the store first," recommended Lew.