Given Google’s recent social efforts (Twitter Search, Social Search, Google Buzz, etc.), I am curious to see what they will do with the Aardvark product – will it stand alone as it has or will it find its way into existing or new Google tools? I, for one, would love to see it integrated into Google’s main search. One consequence of Google’s recent launch of Buzz is reminding people that Google has been collecting data on your social network for a while now. If Aardvark were integrated into your Google network, we’d have a out-of-the-box solution for social search (no messy profile-connecting or friend-inviting needed! To me, it seems like one of the biggest hurdles for most people in terms of social search or networking tools is the cost of building up their networks, so this would provide a quick and easy way around that.

What will GoogVark look like? I can’t say I’ve ever used the “I’m Feeling Lucky” button, but I do know that there are times when I can’t quite find the best answers through Google search, and I’d love to be able to seamlessly shift over to social search. I personally would love to see something like this (with an example supplied by Google itself!):

Would this make social search more inviting to you?

P.S. Congratulations to Aardvark’s founders over at The Mechanical Zoo – you guys deserve it!

The paper, by Aardvark’s Damon Horowitz and Stanford’s Sep Kamvar, focuses mostly on the architecture of the Aardvark system, from the external representations with which users interact to the internal ranking algorithms on which the system runs. Below, I present a short summary of what they report, focusing on the elements I found most interesting:

The Basic Model: Aardvark’s scoring function is similar to PageRank in that both utilize two primary, but somewhat independently considered components: relevance and quality.

• Relevance in the Aardvark model pertains to the probability that a particular user i can answer the given question q based on the identified topics contained in t.
• Quality in the Aardvark model pertains to the overall probability that a user i can return a satisfactory answer to another user j, regardless of the question.

Indexing Topics: Aardvark computes the relevance score by calculating a distribution of knowledge over topics known by the user using the following sources (keyword-y sounding italicized terms are for convenience only and are not used in the paper):

• Explicit Prompting at sign-up for three “starter” topics about which the user has expertise.
• Social Prompting of a user’s friends to provide topics about which they trust the user’s opinion.
• Structured Parsing of the online profile pages connected to Aardvark by the user (e.g. “Interests” on a Facebook profile).
• Unstructured Parsing of the users’ online homepage, blog, or status updates using a linear SVM to extract overall subject area and a named entity extractor to extract more specific topics.

Indexing Connections: Aardvark computes the quality score by building a set of weighted connections between users using characteristics ranging from social proximity to similarities in demographics or behavior, such as:

• Social Connections either in the form of explicitly defined “friend” connections or implicit “network” connections, such as both being part of the Stanford network.
• Demographic Similarity, which likely includes age, gender, and location based on profile information collected by Aardvark.
• Profile Similarity, which seems to include similar movies and other items which might be listed on other profiles, such as Facebook.
• Vocabulary Match, which they explain with the example of “IM Shortcuts” (i.e. I assume this means it is based on the language you use to interact with Aardvark, but I am unsure.)
• Chattiness and Verbosity Match, which relate to frequency and length of messages used when interacting with Aardvark.
• Politeness Match, which basically seems to mean whether or not say “Thanks!” or not.
• Speed Match, which is a measure of responsiveness to other users.

Analyzing Questions: While all of the other components are pre-computed, this part is computed at question time (obviously). The utilize a number of classifiers to classify the question and then a set of mappers to map the question to a set of topics, noting that “the role of the Question Analyzer…is simply to learn enough about the qeustion that it may be sent to appropriately interested and knowledgeable human answerers”. Here are the classifiers they list (with the names used in the paper):

• NonQuestionClassifier: Determines if input is a valid question.
• InappropriateQuestionClassifier: Determines if input is obscene, spam, or otherwise unsuitable for asking.
• TrivialQuestionClassifier: Determines if input is a simple factual question (examples given: “What time is it now?”, “What is the weather?”). If so, the user gets an automatically generated answer via traditional web search.
• LocationSensitiveClassifier: Determines if the question contains location information; if it does, it passes that information along to the Routing Engine

• KeywordMatchTopicMapper: Checks for string matches against user profile topics (the mapper attempts to classify meaningful vs. spurious matches).
• TaxonomyTopicMapper: Classifies question text using an SVM trained on an “annotated corpus of several million questions” (where did they find that?)
• SalientTermTopicMapper: Extracts salient phrases using a noun-phrase chunker and tf-idf and finds “semantically similar user topics”.
• UserTagTopicMapper:Utilizes tags explicitly provided by the asker or other answerers and maps them to user topics.

This description of the routing algorithm comprises the main function of the paper. After some more description of how users interact with the system, the authors provide some interesting data collected over the past several months of use (from the beta launch in March 2009 until October 2009).  Here’s a quick run-down of the more interesting facts that they presented:

• Strong User Growth: As of October 2009, they reported 90,361 user accounts, and users appear to be remaining active (in the study period, over 1/2 the users actively generated content and over 2/3 of the users passively participated).

User Growth on Aardvark (graph taken from the paper).

• Higher Query Contextualization: Aardvark queries average 18.6 words in length while the average query length reported for web search is between 2.2 and 2.9 words (citing previous comparison and characterization studies).  They further state that “98.1% of questions are unique”, though I am unsure as to how exact they are being about matching (I am sure the question “What’s a great restaurant in SF” has been asked 1000 times in different forms). In addition, they report from manual scoring of 1000 randomly selected questions that 64.7% of questions asked have a subjective element, with advice about travel, restaurants, and products being specifically popular.
• Fast, High-Quality Answers: They report that 87.7% of questions get answers and 57.2% received an answer within 10 minutes. They report that 70.4% of answers receiving feedback are rated as “good” and only 15.5% are rated as “bad”. Interestingly, they observe a notable difference in feedback on answers from users within the asker’s social network (76% rated as food) and outside the asker’s network (68% rated as good).

Questions on Aardvark (chart taken from the paper).

Overall, I really enjoyed reading this paper. After using Aardvark for over a year now, it was really interesting to get to peer inside and see how the system works, and a lot of great details were provided about the ranking engine.

One place where I feel that the authors missed the mark was in the cursory side-by-side evaluation which pitted Aardvark against Google for a set of 200 questions randomly selected from the Aardvark system. They report that 71.5% of the questions studied were answered successfully on Aardvark, while 70.5% of the questions were answered successfully on Google. This comparison seems mostly useless as the questions, having been pulled from the Aardvark system in the first place, are ones that were specifically chosen because they are better adapted to what is being called ‘social search’. This comparison left me desirous of more investigation into two main questions.

“What makes a search engine ‘social’ in the first place?”

The distinction between social and non-social is extremely murky, something Brynn and I discovered when working on our Social Search paper. It has been argued before (one small example here) that Google’s PageRank algorithm is inherently social, as it aggregates information provided by people (links to one another) to rank results. However, it is clear that something seems categorically different between Google and what people perceive to be ‘social search’. When it comes down to it, even though everyone is excited about Google’s forays into “Social Search”, there’s nothing all that fundamentally different about Google indexing your blog and your tweets than any other documents extant on the web.

To me, it seems that the key difference is really the change in the direction of interaction. While Google takes a query (question) and compares it against traces of discussion about that question from the past (web documents), systems perceived as ‘social’ take a question and attempt to generate new answers in the future. This change in direction is what allows for the higher context that makes ‘social’ search answers so much more rich (at least for some questions.)  Perhaps we need a different word to define this phenomenon – ‘real-time search’ seems to get at it more, but has its own problems.  Perhaps something like ‘generative search’? I really don’t know.

“Why do we need a social search engine at all?”

This one seems like the best fodder for a follow-up study by Aardvark. While they do provide a rough breakdown of the types of questions asked on Aardvark (see pie chart above), I think that a comparison might have been much more interesting if they had looked at a variety of classes of user needs and had compared the relative efficacy of searching on Aardvark and a traditional search engine such as Google. It is clear that ‘social’ will work much better for some needs and much worse for others, but up to this point, people who talk about social search always seem to use the same types of examples (travel, restaurants, and products, for instance). It would be great to get a clear idea over a wide range of needs and use cases where systems such as Aardvark can provide benefits over existing tools.

Anyways, for those of you interested in ‘social search’ and search systems, I encourage you to read this paper and tell me your thoughts!

STATS: He started off with some data about internet usage (many pulled from here), most notably mentioning that there are currently approximately 1.7B Internet users in the world (also adding that as Google’s ‘Internet Evangelist’, he “still has 75% of the world left to go.”).  What was interesting was his focus on what the numbers really meant – while Asia only has an 18.5% Internet penetration rate, that still works out to about 704M people, which is still more than 2x the entire US population.  In addition, he mentioned that there are currently 4 billion mobile phones, a fact which was new to me, and which implied that most people were using mobile phones as their primary conduit to the Internet.

Major Changes: He also shared some of the major changes that were happening soon.  One of these was about the Internet’s current switch to IPv6, mentioning that we were on track to exhaust the stock of IPv4 addresses by 2011 (perhaps sooner if there is a rush for addresses at the end).  He mentioned that the new 128-bit addresses would allow for 3.8 x 10^34 addresses – “a number only Congress can appreciate”.  In addition, he mentioned upcoming changes like internationalization of domain names (allowing non-latin characters in top-level domain names).

Applications: He then did a whirlwind tour of the kinds of applications that are supported by the Internet (mostly seen through Google’s eyes, of course).  Email (GMAIL), Video-Sharing (YOUTUBE), Maps (GMAPS), you get the picture…It was interesting to hear him talk about Google Wave, because even though he spoke about it with conviction, I noticed that it was still difficult for him to really express the use cases for the service, something that I (and many other people) have had a bit of trouble with.

New Types of Devices: He talked about the grand proliferation of internet-enabled things.  This is a topic that I’ve long had some interest in and have been desirous to get involved with (beyond playing around with my Nabaztag, of course).  He actually got into an interesting anecdote about installing temperature sensors in his wine cellar that text messaged him when the temperature rose above 60 degrees.  He talked about possible additions to this project, including actuators to turn on the cooling system remotely, and RFID-indexing all of the bottles to keep inventory.  This transitioned into discussion of putting sensors in the corks themselves to deliver information about the wine, which reminded me a great deal of Bruce Sterling’s discussion of “spimes” in his book, “Shaping Things” (a quick, interesting read, btw – definitely recommended).

Challenges of the Digital Age: Finally (skipping over some things), he talked about 2 big challenges that he saw for information handling in the future.  The first was the phenomenon of ‘Bit Rot’ – he applied the term not to the decay of physical storage media, but rather to the idea that we might just stop deciding to update programs, and thus 1000 years from now, we’ll never be able to see that Powerpoint presentation from 2004.  The other grand challenge he brought up (something he has been heavily involved with) was the creation of the “Interplanetary Internet“.  The idea (linking things in space to the Internet) is relatively simple, but the execution (as one might expect) is somewhat hard.  Challenges such as vast distances (speed of light actually becomes a factor when Mars gets as far as 235 million miles from Earth) and planetary rotation (now you see it, now you don’t), have led them to develop a Delay-Tolerant Network protocol that uses a “store and forward” approach instead of trying to achieve end-to-end communication.  Their plan is to use as nodes satellites which have been re-purposed and re-programmed after completing their original mission.

Anyways, the talk, which covered everything from mobile phones to cloud-computing to internet-enabled surfboards to Googling in space, was interesting and inspiring.  In case you want to watch it, PARC has started putting up videos of the PARC Forum talks, and you can find the Vint Cerf talk here a few days hence.

Here’s a quick list of the papers mentioned:

• “How Medical Expertise Influences Web Search Interaction” [1] and “Characterizing the Influence of Domain Expertise on Web Search Behavior” [2] by Ryen White, Sue Dumais, and Jaime Teevan.  This poster abstract and longer paper present a large-scale, log-based analysis of web searches in 4 domains (Medicine, Finance, Law, and Computer Science), looking specifically at how domain experts differ from non-domain experts in terms of search behavior.  The data for the study were extensive, comprised of a sample of URL visits from users of a browser toolbar over the course of a 3-month period and representing “more than 10 billion URL visits from more than 500 thousand unique users.”
• “Knowledge in the Head and on the Web: Using Topic Expertise to Aid Search” [3] by Geoffrey Duggan and Stephen Payne.  This paper looks at the role of domain expertise in predicting search performance for people searching within their domain of expertise.  The study involved asking 34 university students trivia questions on two topics – Football (they meant to write “Soccer”) and Pop Music – and asking them to answer, first using only their own knowledge, and then again with the help of the Internet.
• “Web search behavior of Internet experts and newbies” [4] by Christoph Holscher and Gerhard Strube.  This is a somewhat earlier paper focusing on identifying the search strategies of internet (search) experts, and then using that ifnromation to help compare the effects of serach expertise and domain expertise on search performance.  In the first study, they had 12 internet experts first do a mental walk-through of their search strategies and then carry out real search tasks using a teach-aloud/think-aloud sort of protocol.  In the second study, they had 24 university students conduct web-based search tasks pertaining to economics – they were divided by domain expertise (half were economics students) and search expertise (assessed by interview and pre-test).
• “Domain knowledge, search behaviour, and search effectiveness of engineering and science students: an exploratory study” [5] by Xiangmin Zhang, Hermina G.B. Anghelescu, Xiaojun Yuan.  This paper examined the relationships connecting domain knwoledge, search behavior, and search effectiveness.  The study established the domain knowledge of 22 engineering studies through familiarity with terms from an engineering thesaurus, and then had them search on 3 assigned topics.

The papers that focused on quantifying search behavior showed that domain experts tended to do more exploration overall than domain novices.  [2] found that they issued more queries, they branched more (branching defined as stepping back to a previous page and then moving forward to a new page), they visited a larger number of unique domains, and they spent a longer time overall per search tasks.  [5] also found that domain experts tended to issue more queries (34.64 queries/subject vs. 20.09) over the course of their search tasks.  In addition, the domain experts explored this space faster: [3] found that greater topic expertise led to less time spent per page and to faster decisions about whether or not to stop a line of inquiry, a finding corroborated by [4].

One interesting point of disagreement involved the length of queries.  [2] pointed to past literature demonstrating that domain experts tended to issue longer queries and more technical query terms, a result replicated in their study.  Longer queries were seen from domain experts in [5], as well (4 terms/query vs. 2.86).  [3], however, found that the domain experts studied used shorter queries than the domain novices, contradicting these other studies (though the scope of consideration was restricted greatly to come to this conclusion – they looked at just 2 of the football questions).  In the first study from [4], when the search behavior of the internet experts was compared against search logs from the Fireball Search Engine, it was found that the internet experts use longer queries (3.64 vs. 1.66 words), but in the second study, those with domain knowledge were found to make shorter queries than those without (1.97 vs. 2.96 words).

The papers that attempted to highlight specific search strategies also revealed some interesting differences.  [2] examined the domain suffixes of the sites visited, and noticed that domain experts tend to visit different types of sites than domain novices.  Computer Science experts, for instance, were more likely to visit *.org or *.edu sites than novices, while novices were more likely to visit *.com sites, representing that experts might be more familiar with academic or industry sites, while novices might be more familiar with consumer-oriented commercial sites.  [4] also found that so-called “double experts” (those with Internet AND domain expertise) tended to navigate directly to “go-to” sources of information, while all other groups were more likely to start with search engines.

Regarding overall performance, it is perhaps not surprising that domain experts performed better in all search tasks.  [3] attempts to distinguish between searching within one’s domain for information already known vs. searching within one’s domain for information that one doesn’t already know, and found that domain experts perform better in both scenarios.  Because [2] did not control user tasks, they coded successes as logged searches where the final click was a URL and failures as when the final click was another search.  Given this coding, they found that experts were more successful than novices when searching in-domain, but that these same experts performed the same as novices when searching for information out of their domain of expertise, highlighting the difference between domain expertise and search expertise.

Some of the interesting questions that come out of this field of research relate to how we can transfer the advantage that domain (and search) experts have to novices.  One possible method is to pin down what these experts are doing that helped them perform better and attempt to work these strategies into instruction.  Obviously, Internet search skills are already immensely important, and I would hope that this would trickle down into educational curricula (if they haven’t already).

Domain experts find information faster because their expertise in the space allows them to identify relevant information faster and to build off of it.  But, for those of us who are attempting to use the web to learn things on our own, there is a serious boot-strapping problem here.  As someone who is mostly self-taught when it comes to programming, I know how difficult it is to face the problem of searching for information when you are not entirely sure what to search for.  Once you get over the initial learning curve, it becomes much easier.  For those of us interested in building new technologies, here is a challenge: How can we create tools that support domain novices by doing this bootstrapping for them?

If we can find a way to identify domain novices and present them with useful information such as definitions or important “go-to” sources, we can significantly speed up their learning so that they can more quickly fend for themselves.  Faceted search tools such as MrTaggy take a solid step towards tackling this problem by providing searchers with additional cues that provide context.  More detailed study is needed regarding how to connect the gaps in which domain novices get lost – as better tools become available for providing socially or computationally-derived contextual data, it will be interesting to see what technologies evolve to support these needs.

Links for Papers Above (Most from ACM Digital Library):

• [1] “How Medical Expertise Influences Web Search Interaction” by Ryen W. White, Susan Dumais, and Jaime Teevan in Special Interest Group on Information Retrieval (SIGIR) 2008.
• [2] “Characterizing the Influence of Domain Expertise on Web Search Behavior” by Ryen W. White, Susan Dumais, and Jaime Teevan in Conference on Web Search and Data Mining (WSDM) 2009.
• [3] “Knowledge in the Head and on the Web: Using Topic Expertise to Aid Search” by Geoffrey B. Duggan and Stephen J. Payne in Conference on Human Factors in Computing Systems (CHI) 2008.
• [4] “Web search behavior of Internet experts and newbies” by Christoph Holscher and Gerhard Strube in Computer Networks 33 (2000), pp.337-346.
• [5] “Domain knowledge, search behaviour, and search effectiveness of engineering and science students: an exploratory study” by Xiangmin Zhang, Hermina G.B. Anghelescu, and Xiaojun Yuan in Information Research 10(2), Jan 2005.

Some Other Reading on this Topic:

• “The Effects of Topic Familiarity on Information Search” by Diane Kelly, Colleen Cool in Joint Conference on Digital Libraries (JCDL) 2002.
• “Domain-Specific Search Strategies for the Effective Retrieval of Healthcare and Shopping Information” by Suresh K. Bhavnani in Conference on Human Factors in Computing Systems (CHI) 2002.
• “The Effects of Domain Knowledge on Search Tactic Formulation” by Barbara M. Wildermuth in Journal of the American Society for Information Science ant Technology, 2004.

Today we’re introducing Raindrop, an exploration in messaging innovation being led by the team responsible for Thunderbird, to explore new ways to use Open Web technologies to create useful, compelling messaging experiences.

We hope to lead and spur the development of extensible applications that help users easily and enjoyably manage their conversations, notifications, and messages across a variety of online services. A central principle behind Raindrop is that messaging should be personal — we want Raindrop to be people-centric both in how we process messages, and in how we can help give people control over their personal data and experiences.

When a friend’s link from YouTube or flickr arrives, your messaging client should be able to show the video or photos near or as part of the message, rather than rudely kicking you over to a separate browser tab. Notifications from computers and mailing lists should be organized for you, not clutter your Inbox or require tedious manual filter setup. It should be easy to smoothly integrate new web services into your conversation viewer entirely using open web technologies.

The post doesn’t remains a little too vague to offer a specific vision of what they are talking about.  Essentially, it sounds like Raindrop will be some sort of aggregator for conversation on the web, delivering messages to you in an email-like format.  The “fundamental ideas” video shines the light a little bit more on the idea of intelligently culling “personal” messages (as opposed to bulk) from your various streams. (P.S. The video didn’t play correctly for me, but you can watch it in large-format at Vimeo here).

The "Second Iteration" of the Raindrop Interface

As I have obviously not yet had the opportunity to try out Raindrop, I can’t really give any sort of review of the service.  However, I think that the design principles here are interesting; with the increasing number of conversation platforms appearing on the web, the need for intelligent aggregation is growing quickly.  Even Friendfeed, the leading social web aggregator, felt unmanageable to me at times, and they weren’t even trying to deal with email!

I also enjoyed the use of “Raindrop” as a name, as it conjured up (for me) a very specific comparison with another possibly water-themed product.  While Google Wave’s approach to aggregating information is to literally inundate you with it and force you to use the search function to paddle your way out, Raindrop (in theory, at least) seems to focus on keeping messages separate, allowing you to catch a few in your hand when you need them.