Based on this post and other comments, it is fairly clear that MG Siegler is on “Team Quora”, calling Quora “a great source of information like Twitter and Facebook and blogs themselves.” Having been a Quora user for several months, I have found the quality of answers on the site to be extremely high. These have ranged from the much-celebrated cases of high-profile individuals answering questions about topics pertaining to them (e.g. Netflix CEO Reed Hastings answering the question “How much does Netflix spend on postage every year?“) to the opportunities for creative individuals to answer questions in awesome and innovative ways (see Wavii programmer Erik Frey‘s answer to the question “Which animal has been used most frequently for a band name?“). I have also personally asked a number of questions and gotten timely and high-quality answers.

However, the key question here is whether the site will continue to be as useful as more and more people join. While many might argue that including more subject matter experts can only improve the site, one must also remember that this increased signal is only useful when it can be separated from the increased noise. Right now, Quora is a bit like <nerd alert>Flynn’s cave dwelling in the Outlands, where the information contained is only safe as long as the masses can’t get to it</nerd alert>; users can trust the answers they find because they often come paired with often famous or at least recognizable names and faces. Yahoo! Answers is a great example of how a Q&A site can decrease in quality with respect to both the questions and answers as it opens up (for a quick, possibly NSFWish laugh, check out “11 Stupid Questions from Yahoo Answers That Have Changed My Life“) Even if you develop the most robust social answer-quality-checker imaginable, the presence of thousands of stupid questions tagged with topic tags that direct them to your inbox is going to turn a lot of the quality answer providers away from the site. In some ways, I felt that Aardvark fell into that trap as it grew more popular, and I now find myself answering a lot of questions that involved identifying rashes.

I had this question about the possible perils of mainstream adoption in mind, when reading a NY Times blog post this week, in which David Pogue describes his first interactions with the site as “a descent into bafflement.” Among the parts of the site that he deems confusing are the login process, the task of adding connections and following topics, and the actual task of asking a question. For chronicling his confusion, however, he earned the following response from Siegler (on his personal blog):

Is this meant to be written from the perspective of a 95-year-old senile man?

Apparently, every site should be designed in a way so that it’s just like every other site that failed before it on the Internet. Makes perfect sense.

Prediction: he’ll love Quora in 12 months.

Now, I would consider David Pogue to be a relatively tech-savvy individual. He’s been blogging about Web/technology topics for over 10 years now, and I generally find his posts to be pretty interesting and insightful. If he is having this much difficulty using the site, I think that his frustrations really do say something about Quora’s current potential to reach beyond the geek crowd into the general public . And on some level, I think that the desire to mock him for this reflects an underlying recognition that the Silicon Valley influentials who currently use the site wouldn’t actually benefit from “regular people” using it, since this could potentially spell the end of Quora’s usefulness. Perhaps a better use of time might be to consider how to make the site more user-friendly and how to maintain the quality as new users arrive.

Studying sites like Wikipedia is a great way to examine how to maintain quality while scaling out to a broader audience. While it’s difficult to ascertain what actually makes Wikipedia work (in practice, if not in theory), many of the qualities inherent in Wikipedia are those identified in a 2005 paper [1] by J.M. Leimeister and colleagues at the Technische Universität München as factors which promote community success in an environment where trusted information is key. Some of these include exposing the identity of content providers, clearly establishing goals for the community, making member profiles available to other members, and providing various levels of anonymity, all of which are things which are built into Wikipedia’s core. Another feature recognized across the literature is recognition and rewards for contributors, something embodied in Wikipedia in the form of “barnstars“. Quora does a great job of exposing identity (which will likely make it much better than Yahoo! Answers), but I don’t believe that it adequately addresses these other elements. I think that the addition of “moderators” or other custodian roles for hyper-motivated users could be the kind of thing that keeps the Quora community in check, and I would be eager to see them roll something like that out before opening the site up to the general public (I think it’s technically still in beta, no?)

Quora is a great source of information for me in some of the same ways that Facebook and Twitter are. The key difference is that in those media, I can control what I see and who it comes from. If you want to imagine the utility of those sites without such controls, just imagine trying to sift through the real-time Twitter stream – it’s mind-numbing, to be kind (unless of course you are a Justin Bieber fan, in which case you should be delighted). The goals of those who currently enjoy Quora’s usefulness should be to help maintain that usefulness as the site grows. I’m not saying that this will be an easy task, but if it isn’t accomplished, I don’t think that anyone is going to love the site in 12 months.

[1] Leimeister, J.M., Ebner, W., & Krcmar, H. 2005: Design, implementation, and evaluation of trust-supporting components in virtual communities for patients. Journal of Management Information Systems 21, 4, 101-135.

Given Google’s recent social efforts (Twitter Search, Social Search, Google Buzz, etc.), I am curious to see what they will do with the Aardvark product – will it stand alone as it has or will it find its way into existing or new Google tools? I, for one, would love to see it integrated into Google’s main search. One consequence of Google’s recent launch of Buzz is reminding people that Google has been collecting data on your social network for a while now. If Aardvark were integrated into your Google network, we’d have a out-of-the-box solution for social search (no messy profile-connecting or friend-inviting needed! To me, it seems like one of the biggest hurdles for most people in terms of social search or networking tools is the cost of building up their networks, so this would provide a quick and easy way around that.

What will GoogVark look like? I can’t say I’ve ever used the “I’m Feeling Lucky” button, but I do know that there are times when I can’t quite find the best answers through Google search, and I’d love to be able to seamlessly shift over to social search. I personally would love to see something like this (with an example supplied by Google itself!):

Would this make social search more inviting to you?

P.S. Congratulations to Aardvark’s founders over at The Mechanical Zoo – you guys deserve it!

The paper, by Aardvark’s Damon Horowitz and Stanford’s Sep Kamvar, focuses mostly on the architecture of the Aardvark system, from the external representations with which users interact to the internal ranking algorithms on which the system runs. Below, I present a short summary of what they report, focusing on the elements I found most interesting:

The Basic Model: Aardvark’s scoring function is similar to PageRank in that both utilize two primary, but somewhat independently considered components: relevance and quality.

• Relevance in the Aardvark model pertains to the probability that a particular user i can answer the given question q based on the identified topics contained in t.
• Quality in the Aardvark model pertains to the overall probability that a user i can return a satisfactory answer to another user j, regardless of the question.

Indexing Topics: Aardvark computes the relevance score by calculating a distribution of knowledge over topics known by the user using the following sources (keyword-y sounding italicized terms are for convenience only and are not used in the paper):

• Explicit Prompting at sign-up for three “starter” topics about which the user has expertise.
• Social Prompting of a user’s friends to provide topics about which they trust the user’s opinion.
• Structured Parsing of the online profile pages connected to Aardvark by the user (e.g. “Interests” on a Facebook profile).
• Unstructured Parsing of the users’ online homepage, blog, or status updates using a linear SVM to extract overall subject area and a named entity extractor to extract more specific topics.

Indexing Connections: Aardvark computes the quality score by building a set of weighted connections between users using characteristics ranging from social proximity to similarities in demographics or behavior, such as:

• Social Connections either in the form of explicitly defined “friend” connections or implicit “network” connections, such as both being part of the Stanford network.
• Demographic Similarity, which likely includes age, gender, and location based on profile information collected by Aardvark.
• Profile Similarity, which seems to include similar movies and other items which might be listed on other profiles, such as Facebook.
• Vocabulary Match, which they explain with the example of “IM Shortcuts” (i.e. I assume this means it is based on the language you use to interact with Aardvark, but I am unsure.)
• Chattiness and Verbosity Match, which relate to frequency and length of messages used when interacting with Aardvark.
• Politeness Match, which basically seems to mean whether or not say “Thanks!” or not.
• Speed Match, which is a measure of responsiveness to other users.

Analyzing Questions: While all of the other components are pre-computed, this part is computed at question time (obviously). The utilize a number of classifiers to classify the question and then a set of mappers to map the question to a set of topics, noting that “the role of the Question Analyzer…is simply to learn enough about the qeustion that it may be sent to appropriately interested and knowledgeable human answerers”. Here are the classifiers they list (with the names used in the paper):

• NonQuestionClassifier: Determines if input is a valid question.
• InappropriateQuestionClassifier: Determines if input is obscene, spam, or otherwise unsuitable for asking.
• TrivialQuestionClassifier: Determines if input is a simple factual question (examples given: “What time is it now?”, “What is the weather?”). If so, the user gets an automatically generated answer via traditional web search.
• LocationSensitiveClassifier: Determines if the question contains location information; if it does, it passes that information along to the Routing Engine

• KeywordMatchTopicMapper: Checks for string matches against user profile topics (the mapper attempts to classify meaningful vs. spurious matches).
• TaxonomyTopicMapper: Classifies question text using an SVM trained on an “annotated corpus of several million questions” (where did they find that?)
• SalientTermTopicMapper: Extracts salient phrases using a noun-phrase chunker and tf-idf and finds “semantically similar user topics”.
• UserTagTopicMapper:Utilizes tags explicitly provided by the asker or other answerers and maps them to user topics.

This description of the routing algorithm comprises the main function of the paper. After some more description of how users interact with the system, the authors provide some interesting data collected over the past several months of use (from the beta launch in March 2009 until October 2009).  Here’s a quick run-down of the more interesting facts that they presented:

• Strong User Growth: As of October 2009, they reported 90,361 user accounts, and users appear to be remaining active (in the study period, over 1/2 the users actively generated content and over 2/3 of the users passively participated).

User Growth on Aardvark (graph taken from the paper).

• Higher Query Contextualization: Aardvark queries average 18.6 words in length while the average query length reported for web search is between 2.2 and 2.9 words (citing previous comparison and characterization studies).  They further state that “98.1% of questions are unique”, though I am unsure as to how exact they are being about matching (I am sure the question “What’s a great restaurant in SF” has been asked 1000 times in different forms). In addition, they report from manual scoring of 1000 randomly selected questions that 64.7% of questions asked have a subjective element, with advice about travel, restaurants, and products being specifically popular.
• Fast, High-Quality Answers: They report that 87.7% of questions get answers and 57.2% received an answer within 10 minutes. They report that 70.4% of answers receiving feedback are rated as “good” and only 15.5% are rated as “bad”. Interestingly, they observe a notable difference in feedback on answers from users within the asker’s social network (76% rated as food) and outside the asker’s network (68% rated as good).

Questions on Aardvark (chart taken from the paper).

Overall, I really enjoyed reading this paper. After using Aardvark for over a year now, it was really interesting to get to peer inside and see how the system works, and a lot of great details were provided about the ranking engine.

One place where I feel that the authors missed the mark was in the cursory side-by-side evaluation which pitted Aardvark against Google for a set of 200 questions randomly selected from the Aardvark system. They report that 71.5% of the questions studied were answered successfully on Aardvark, while 70.5% of the questions were answered successfully on Google. This comparison seems mostly useless as the questions, having been pulled from the Aardvark system in the first place, are ones that were specifically chosen because they are better adapted to what is being called ‘social search’. This comparison left me desirous of more investigation into two main questions.

“What makes a search engine ‘social’ in the first place?”

The distinction between social and non-social is extremely murky, something Brynn and I discovered when working on our Social Search paper. It has been argued before (one small example here) that Google’s PageRank algorithm is inherently social, as it aggregates information provided by people (links to one another) to rank results. However, it is clear that something seems categorically different between Google and what people perceive to be ‘social search’. When it comes down to it, even though everyone is excited about Google’s forays into “Social Search”, there’s nothing all that fundamentally different about Google indexing your blog and your tweets than any other documents extant on the web.

To me, it seems that the key difference is really the change in the direction of interaction. While Google takes a query (question) and compares it against traces of discussion about that question from the past (web documents), systems perceived as ‘social’ take a question and attempt to generate new answers in the future. This change in direction is what allows for the higher context that makes ‘social’ search answers so much more rich (at least for some questions.)  Perhaps we need a different word to define this phenomenon – ‘real-time search’ seems to get at it more, but has its own problems.  Perhaps something like ‘generative search’? I really don’t know.

“Why do we need a social search engine at all?”

This one seems like the best fodder for a follow-up study by Aardvark. While they do provide a rough breakdown of the types of questions asked on Aardvark (see pie chart above), I think that a comparison might have been much more interesting if they had looked at a variety of classes of user needs and had compared the relative efficacy of searching on Aardvark and a traditional search engine such as Google. It is clear that ‘social’ will work much better for some needs and much worse for others, but up to this point, people who talk about social search always seem to use the same types of examples (travel, restaurants, and products, for instance). It would be great to get a clear idea over a wide range of needs and use cases where systems such as Aardvark can provide benefits over existing tools.

Anyways, for those of you interested in ‘social search’ and search systems, I encourage you to read this paper and tell me your thoughts!

I purposely kept the subject “Social Search” vague, because I hoped to cast a wider net and then decide who to follow more specifically later.  It’s turned out that I’ve already learned a ton of interesting stuff from the people here, so if you are interested in Social Search topics at all, I’d recommend that you join the list and give everyone else a follow!

I had trouble embedding the spreadsheet in the post, so click these links for the sign-up form and results list. Enjoy!

P.S. The list is obviously public as it was passed around on Twitter, but if anybody has any questions or concerns about my posting the list here, please let me know!