Fri, 29 Jun 2012 08:23:20 PDT
In what should not be a surprise given Siri's beta status and its use of new
technology for voice interaction with mobile devices, a study from Piper
Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed
by the traditional text-entry Google searches that have been in place for
years.
The study included presenting Siri with 800 queries in each of two
environments, a quiet room and a busy urban street, and then assessing Siri'
s ability to correctly interpret the queries and provide correct responses.
Specifically addressing the outdoor testing, Munster notes:
While Google comprehends 100% of searches, Siri was only able to comprehend
83% of the prompted questions on a busy Minneapolis street. In terms of
answer accuracy, Siri correctly answered 62% of queries on the street
compared to Google's estimated 86% answer accuracy. Based on these results,
we give Google an A+ for comprehension and a B+ for accuracy while we give
Siri a B for comprehension and a D for accuracy.
But while Siri has some room for improvement, Munster acknowledges that the
technology is only in its infancy and he remains "optimistic about its
future" with an eye toward iOS 6 as bringing substantial improvements.
Piper Jaffray's testing was conducted using the iPhone's built-in microphone
in the quiet setting and Skullcandy microphone headphones for the outdoor
setting, with Siri's ability to correctly comprehend queries falling from 89
% to 83% when moving outdoors to the noisy urban setting. On the accuracy
side, Google's 86% rating is derived from comScore data showing that Google
generates 1.14 search result pages per search, suggesting that roughly 86%
of time Google presents the data the user is looking for within the first
few results.
Piper Jaffray also analyzed the sources for Siri's results, finding that
Google provided 60% of the answers, with Yelp and other sources filling in
the remainder.
Of queries excluding commands (i.e. call Jay, send text) Google would
provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and
Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of
navigation results, 61% of information results, 48% of commerce results and
42% of local results. Among other result aggregators, Yelp provided the
most local results (51%) and commerce results (51%), while WolframAlpha
provided 34% of information results.
Munster estimates that Google's share of Siri results will drop to 48% with
the launch of iOS 6 as navigation, sports, and movie queries shift over to
Apple and its new partners.
