We believe experiences matter.™
Eye Tracking Bing vs. Google: A First Look
User Centric, Inc., a user research firm based in Chicago, offers a glimpse into the battle between the newly launched Microsoft's Bing and the powerful incumbent, Google. Eye tracking technology was used to capture 21 participants' eye movements as they completed two informational (e.g., "Learn about eating healthy") and two transactional (e.g., "Book a last minute vacation") search tasks in each engine.
The goals of the research were (1) to compare the distribution of attention on equivalent areas of Bing and Google and (2) to assess how much attention is captured by elements that are unique to Bing.
Distribution of attention on Google vs. Bing:
- Google and Bing did not differ in terms of the amount of attention on the organic search results. In each search, all participants looked at the organic search results, spending an average of 7 seconds in that area.
- Attention on the sponsored links located above the organic results was similarly high for both Bing and Google. Over 90% of participants looked in that area during each search. As expected, during transactional searches, participants would spend more time looking at the sponsored results on top (~2.5 seconds) than they did on informational searches (~1.5 seconds).
- However, sponsored links on the right attracted more attention on Bing (~42% of participants per search) than they did on Google (~25% of participants per search). The participants who fixated on these links spent approximately 2.5 seconds looking at the area during transactional searches and 2 seconds during informational searches. These times were similar for the two search engines.
- Another difference between Bing and Google involved related searches. On Bing, related searches are shown on the left, right below the categories, while on Google, related searches are below the organic search results, towards the bottom of the page. Bing's related searches had a much higher visibility than Google's, attracting the attention of 31% of participants per search. Google's related searches attracted the attention of only 5% of participants per search.
User Centric also investigated three additional features that differentiate Bing from Google: Flyouts that appear when users hover over individual search results, a category list displayed on the left, and sponsored links at the bottom of the search results page.
- Most participants (67%) triggered a flyout at least once during the study, which shows that they are easily activated. Nearly all of these activations were categorized as accidental because participant's attention was away from the mouse pointer which triggered the flyout. Usually, the sudden appearance of an element in the visual field attracts attention but this was not the case in this study. Only 14% of participants (a quarter of those who activated at least one flyout) looked at the flyouts. Users' tendency to ignore the flyouts was likely a result of the learned strategy to devalue motion as a source of information on the Web. This behavior may be compared to "banner blindness.
- Bing's categories displayed on the left attracted much more attention than the flyouts. Half of the participants discovered the categories and three participants even selected a category to refine their search. Each of these three participants chose to use the categories on subsequent searches, which suggests potential value.
- The sponsored links at the bottom of the search results page did not generate any attention. None of the 21 participants looked at the bottom links on any of the four searches. This does not mean that users never look at the bottom sponsored links. Rather, the incidence rate of this behavior may be too low to detect with our sample size.
This study represents preliminary research on the user experience associated with the Bing search engine. User Centric plans to further investigate the discoverability of flyouts and their value proposition in users' search behaviors, as well as other sections of Bing, such as Images, Videos, and News.
Read our latest Bing vs. Google study, "Eyes Linger Longer with Google than Bing." (1/27/2011)
To access additional eye tracking articles and learn more about User Centric's eye tracking services, visit www.UserCentric.com/eyetracking.
Contact Us for Answers
The Handbook of Global User Research collects insight from user experience professionals in over 20 countries. This book is a must-read for professionals working on global research projects. More Info
Comments
The explanation
What I'm missing in this article is the pretty obvious explanation: due to layout with the left-hand navigation of Bing, the natural results are more towards the centre of the screen than on Google. So the distance between the natural results and the ads is less. So it's pretty obvious more people will notice the ads. Don't need eyetracking to know that, although eyetracking confirms what every usability expert would have 'guessed'.
Thanks for your comment
We also hypothesized that Bing's right rail would attract more attention than Google's. As you noted, one of the reasons for this may be the fact that there is less separation between Bing's organic results in the center and the sponsored ads on the right.
Another likely explanation is the novelty effect. Our participants had not used Bing prior to the study but all were very familiar with Google. If you go to a site that you have never been to, you will scan more areas than you normally would if you were familiar with the site because you are trying to orient yourself.
We discussed these and other possible reasons in an interview with PaidContent.org.
We are planning to conduct a follow-up study with Bing users as our participants, so we can determine if the increased amount of attention on the right rail in our first study was due to Bing’s newness or the layout.
Quantitative data is always better!
It may seem obvious to any "usability expert," but it is important to quantify the results. Sometimes opinions do not carry sufficient weight in business decisions. Data matter more!
Distance, screen resolution
Anecdotally, I've noticed similar results to those seen above with the Google heatmap through user testing. I postulated that too much white space or distance from organic results is the reason and as screen resolutions increase, this distance only widens.
nice eye tracking study
Interesting to see more of these, and thank you, it's great to show clients how people can typically act and why you need to be ranked high and how running adwords with good rankings can help Branding and likely CTR.
interesting alternate study
... a few more details and a bit more narrative. Also interesting:
http://www.techcrunch.com/2009/06/25/study-suggests-people-prefer-bings-...
High res screen design
Thank you Usercentric team - you did great and essential study. Thanks for sharing it with us. I think that Bing design's success is because Bing is designed for wide screen, while Google tries (and does it pretty well) to design simple interfaces suitable for almost any device. Anyways, the level of ad's visibility is not everything. Does Bing serve as relevant ads as Google does? Do they filter well enough inappropriate ads?
why eye tracking?
At first, I was expecting this to be obvious, but then considering the novelty effect, I am willing to accept Bing did well for its age. Maybe because MS marketed it well than they did Live Search. I guess Bing's USP was their related search being laid out one below the other as opposed to an alternate stupid design of laying them next to each other, like I've seen in some search engines. I think it occurs in one of Microsoft's MSDN searches, maybe I am not so sure then.
changes to google results
how will this data change with Google moving removing some of the space and moving AdWords closer to the organic results. Also with the increasing the space dedicated to sponsored links such as Ebay multiple sitelinks and sponsored Local results links?
Attention
I think I could gaze at a results screen and not accomplish anything nor attend to anything. With those heatmaps... red ain't always good...
If the design goal is to decrease or eliminate this F phenomenon. Eye tracking may help if the site designs were actually different (i.e. browsing small pictures vs. text)
Eyetracking says little about how people interact with these sites.
Thanks for your comment
We agree that heatmaps can be misleading. Heatmaps are purely a visualization of the data. Nothing more. Nothing less.
The data tell the story. The data found in this study showed differences between the two sites in terms of attention to key areas of the site.
Interpretation of why these differences exist is open to speculation, but the differences found were statistically significant. Please consider the data collected and analyzed as the core of the paper with heatmaps as visualizations of the data found.
Statistically significant?
Whilst this is an interesting study it cannot be statistically significant as there were only 21 people in the study and there are billions of internet users. Novelty factor also invalidates any statistical significance as it is a huge "validity threat".
This is a useful anecdote which supports other data which demonstrates Bing has a higher click through rate for paid ads than Google on average. Eyetracking is interesting but unless you test 1,000s of people ultimately just anecdotal evidence and not emperical data.
On statistical significance
Hypothesis testing (i.e., trying to determine if there is a difference) is frequently confused with precision testing (i.e., trying to generalize an exact "score" to the population). This confusion leads to a lot of criticism regarding sample sizes used in research studies.
We do not claim that the paid ads on Bing or Google receive the same amount of attention in real life as the amount we found in our study. To do that we would have to run hundreds of participants.
Our results indicate, however, that there is a significant difference between Google and Bing (at alpha level of 0.05). In other words, Bing and Google will differ (in terms of a certain measure) 19 out of 20 times. Being able to detect a significant difference indicates that the sample size used in the study was sufficient. An insufficient sample size usually results in an inability to detect a difference where the difference really exists rather than in detecting a difference where it doesn’t exist. If a statistically significant difference is found with a small sample size, this indicates that the difference does exist and is potentially even quite large.
I hope this helps!
Eye Tracking Bing vs. Google
I thought the results were a bit different. Does it depend on the market where your business is?
Thanks