Powered by Twitter Tools

Powered by Twitter Tools

Powered by Twitter Tools

Powered by Twitter Tools

THE OBJECT OF THE AUTHOR in the following pages has been to collect the most remarkable instances of those moral epidemics which have been excited, sometimes by one cause and sometimes by another, and to show how easily the masses have been led astray, and how imitative and gregarious men are, even in their infatuations and crimes.

Charles Mackay – Memoirs of Extraordinary Popular Delusions and the Madness of the Crowds

Image courtesy of Marcos Vasconcelos (click for original)

I’ve been interested a bit in looking at how to help people find high-quality information on the web – recently, I have been exploring how to help people make better credibility judgments about the information they find. One paper I was reading, “Statement Map: Assisting Information Credibility Analysis by Visualizing Arguments” by Koji Murakami and others at the Nara Institute of Science and Technology in Japan, uses as a motivating example the recent movement against vaccinations for children, specifically the MMR (Measles, Mumps, and Rubella), as the result of fears that these vaccines could cause autism.

Back in 2003, Pew reported that over 80% of Internet users have searched for health information (such as info about fitness or vaccinations) online, and one can imagine that this number has only grown since then, so this example is important in illustrating the potential impact of such health memes. I’ve heard multiple parents mention the supposed vaccination-autism link before when making decisions about whether or not to vaccinate their children, so it’s extremely important to figure out how to make sure these parents get intelligent, credible information when searching on the web. The Murakami paper provides some interesting background information on how this particular meme first started:

In 1997, a group of researchers in the UK lead (sic) by Dr. Andrew Wakefield published a study implying a causal connection between Measles, Mumps, and Rubella (MMR) vaccinations and the development of autism in children. Though further scrutiny of these initial results  disproved the autism-vaccination link – culminating in the withdrawal of endorsements by 10 of the study’s 12 authors – the damage had already been done.

The consequences of this single, spurious, study have already been far-reaching. The resulting backlash precipitated a drop in vaccination rates in the UK (where the study was first published), which has led to an increase in outbreaks of measles over the past decade to the point where measles are once again (after transmission was halted 14 years ago) being considered endemic. Even vaccination rates here in the Bay Area have dropped, with the Examiner reporting that vaccination rates are as low as 50% for some Bay Area schools.

I suppose a lot of this can be chalked up to mainstream media coverage of the original study (as well as coverage of well-meaning, but misguided celebrity activists like Jenny McCarthy). However, a large part of why this meme has continued even after the original study was shown to be dubious is due to social phenomena such as communal reinforcement (or the “millions of people can’t be wrong” phenomenon) that can occur so easily on the web. Because it is so easy to publish information on the web, and because information published tends to persist, it is easy to find a wealth of documents supporting any viewpoint, no matter how much evidence there actually is to support that claim. In this case, one can read 100 different news articles, blog posts, and other online resources based on the Wakefield, et al. study without knowing first that these stories do not corroborate each other (as they are drawn from the same small, possibly falsified study) and second that the original study was actually recently retracted by the journal in the first place.

The way that these stories snowball and take on a life of their own is something that Charles Mackay documented in his book “Extraordinary Popular Delusions and the Madness of the Crowds” (the first lines of which are quoted above). He tackles a variety of subjects ranging from the Dutch tulip craze of the 16th century (cf. “21st century housing bubble“) to alchemy to witch hunts. From this book (written in 1841), we can see that the often reasonable shortcuts that people make when processing new information can sometimes lead to these self-propagating effects which take on a life of their own.

The unfortunate fact is that just because the web gives us access to more information doesn’t guarantee that we are going to choose and use it wisely. This is why building tools to help people make better credibility judgments online is so important, raising two questions:

• How do we extract data from within a single web page to help people make better judgments about the information it contains? I know that there is a good deal of work on this topic in Wikipedia with tools like Wikipedia Scanner and PARC’s WikiDashboard helping to to expose author and change information, but how can we bring tools like these to the web as a whole?
• How do we connect data across web pages to hep propagate changes in information across the web? As an example, if information about the study’s retraction could be propagated to pages reporting on the study, parents reading those pages would be less likely to be led astray, possibly saving lives.

For those who are specifically interested in the MMR vaccine controversy, the Wikipedia page links to a lot of good resources, including a long list of studies conducted in the last decade which show no link between autism and the vaccine.

Powered by Twitter Tools

Given Google’s recent social efforts (Twitter Search, Social Search, Google Buzz, etc.), I am curious to see what they will do with the Aardvark product – will it stand alone as it has or will it find its way into existing or new Google tools? I, for one, would love to see it integrated into Google’s main search. One consequence of Google’s recent launch of Buzz is reminding people that Google has been collecting data on your social network for a while now. If Aardvark were integrated into your Google network, we’d have a out-of-the-box solution for social search (no messy profile-connecting or friend-inviting needed! To me, it seems like one of the biggest hurdles for most people in terms of social search or networking tools is the cost of building up their networks, so this would provide a quick and easy way around that.

What will GoogVark look like? I can’t say I’ve ever used the “I’m Feeling Lucky” button, but I do know that there are times when I can’t quite find the best answers through Google search, and I’d love to be able to seamlessly shift over to social search. I personally would love to see something like this (with an example supplied by Google itself!):

Would this make social search more inviting to you?

P.S. Congratulations to Aardvark’s founders over at The Mechanical Zoo – you guys deserve it!

Powered by Twitter Tools

The paper, by Aardvark’s Damon Horowitz and Stanford’s Sep Kamvar, focuses mostly on the architecture of the Aardvark system, from the external representations with which users interact to the internal ranking algorithms on which the system runs. Below, I present a short summary of what they report, focusing on the elements I found most interesting:

The Basic Model: Aardvark’s scoring function is similar to PageRank in that both utilize two primary, but somewhat independently considered components: relevance and quality.

• Relevance in the Aardvark model pertains to the probability that a particular user i can answer the given question q based on the identified topics contained in t.
• Quality in the Aardvark model pertains to the overall probability that a user i can return a satisfactory answer to another user j, regardless of the question.

Indexing Topics: Aardvark computes the relevance score by calculating a distribution of knowledge over topics known by the user using the following sources (keyword-y sounding italicized terms are for convenience only and are not used in the paper):

• Explicit Prompting at sign-up for three “starter” topics about which the user has expertise.
• Social Prompting of a user’s friends to provide topics about which they trust the user’s opinion.
• Structured Parsing of the online profile pages connected to Aardvark by the user (e.g. “Interests” on a Facebook profile).
• Unstructured Parsing of the users’ online homepage, blog, or status updates using a linear SVM to extract overall subject area and a named entity extractor to extract more specific topics.

Indexing Connections: Aardvark computes the quality score by building a set of weighted connections between users using characteristics ranging from social proximity to similarities in demographics or behavior, such as:

• Social Connections either in the form of explicitly defined “friend” connections or implicit “network” connections, such as both being part of the Stanford network.
• Demographic Similarity, which likely includes age, gender, and location based on profile information collected by Aardvark.
• Profile Similarity, which seems to include similar movies and other items which might be listed on other profiles, such as Facebook.
• Vocabulary Match, which they explain with the example of “IM Shortcuts” (i.e. I assume this means it is based on the language you use to interact with Aardvark, but I am unsure.)
• Chattiness and Verbosity Match, which relate to frequency and length of messages used when interacting with Aardvark.
• Politeness Match, which basically seems to mean whether or not say “Thanks!” or not.
• Speed Match, which is a measure of responsiveness to other users.

Analyzing Questions: While all of the other components are pre-computed, this part is computed at question time (obviously). The utilize a number of classifiers to classify the question and then a set of mappers to map the question to a set of topics, noting that “the role of the Question Analyzer…is simply to learn enough about the qeustion that it may be sent to appropriately interested and knowledgeable human answerers”. Here are the classifiers they list (with the names used in the paper):

• NonQuestionClassifier: Determines if input is a valid question.
• InappropriateQuestionClassifier: Determines if input is obscene, spam, or otherwise unsuitable for asking.
• TrivialQuestionClassifier: Determines if input is a simple factual question (examples given: “What time is it now?”, “What is the weather?”). If so, the user gets an automatically generated answer via traditional web search.
• LocationSensitiveClassifier: Determines if the question contains location information; if it does, it passes that information along to the Routing Engine

• KeywordMatchTopicMapper: Checks for string matches against user profile topics (the mapper attempts to classify meaningful vs. spurious matches).
• TaxonomyTopicMapper: Classifies question text using an SVM trained on an “annotated corpus of several million questions” (where did they find that?)
• SalientTermTopicMapper: Extracts salient phrases using a noun-phrase chunker and tf-idf and finds “semantically similar user topics”.
• UserTagTopicMapper:Utilizes tags explicitly provided by the asker or other answerers and maps them to user topics.

This description of the routing algorithm comprises the main function of the paper. After some more description of how users interact with the system, the authors provide some interesting data collected over the past several months of use (from the beta launch in March 2009 until October 2009).  Here’s a quick run-down of the more interesting facts that they presented:

• Strong User Growth: As of October 2009, they reported 90,361 user accounts, and users appear to be remaining active (in the study period, over 1/2 the users actively generated content and over 2/3 of the users passively participated).

User Growth on Aardvark (graph taken from the paper).

• Higher Query Contextualization: Aardvark queries average 18.6 words in length while the average query length reported for web search is between 2.2 and 2.9 words (citing previous comparison and characterization studies).  They further state that “98.1% of questions are unique”, though I am unsure as to how exact they are being about matching (I am sure the question “What’s a great restaurant in SF” has been asked 1000 times in different forms). In addition, they report from manual scoring of 1000 randomly selected questions that 64.7% of questions asked have a subjective element, with advice about travel, restaurants, and products being specifically popular.
• Fast, High-Quality Answers: They report that 87.7% of questions get answers and 57.2% received an answer within 10 minutes. They report that 70.4% of answers receiving feedback are rated as “good” and only 15.5% are rated as “bad”. Interestingly, they observe a notable difference in feedback on answers from users within the asker’s social network (76% rated as food) and outside the asker’s network (68% rated as good).

Questions on Aardvark (chart taken from the paper).

Overall, I really enjoyed reading this paper. After using Aardvark for over a year now, it was really interesting to get to peer inside and see how the system works, and a lot of great details were provided about the ranking engine.

One place where I feel that the authors missed the mark was in the cursory side-by-side evaluation which pitted Aardvark against Google for a set of 200 questions randomly selected from the Aardvark system. They report that 71.5% of the questions studied were answered successfully on Aardvark, while 70.5% of the questions were answered successfully on Google. This comparison seems mostly useless as the questions, having been pulled from the Aardvark system in the first place, are ones that were specifically chosen because they are better adapted to what is being called ’social search’. This comparison left me desirous of more investigation into two main questions.

“What makes a search engine ’social’ in the first place?”

The distinction between social and non-social is extremely murky, something Brynn and I discovered when working on our Social Search paper. It has been argued before (one small example here) that Google’s PageRank algorithm is inherently social, as it aggregates information provided by people (links to one another) to rank results. However, it is clear that something seems categorically different between Google and what people perceive to be ’social search’. When it comes down to it, even though everyone is excited about Google’s forays into “Social Search”, there’s nothing all that fundamentally different about Google indexing your blog and your tweets than any other documents extant on the web.

To me, it seems that the key difference is really the change in the direction of interaction. While Google takes a query (question) and compares it against traces of discussion about that question from the past (web documents), systems perceived as ’social’ take a question and attempt to generate new answers in the future. This change in direction is what allows for the higher context that makes ’social’ search answers so much more rich (at least for some questions.)  Perhaps we need a different word to define this phenomenon – ‘real-time search’ seems to get at it more, but has its own problems.  Perhaps something like ‘generative search’? I really don’t know.

“Why do we need a social search engine at all?”

This one seems like the best fodder for a follow-up study by Aardvark. While they do provide a rough breakdown of the types of questions asked on Aardvark (see pie chart above), I think that a comparison might have been much more interesting if they had looked at a variety of classes of user needs and had compared the relative efficacy of searching on Aardvark and a traditional search engine such as Google. It is clear that ’social’ will work much better for some needs and much worse for others, but up to this point, people who talk about social search always seem to use the same types of examples (travel, restaurants, and products, for instance). It would be great to get a clear idea over a wide range of needs and use cases where systems such as Aardvark can provide benefits over existing tools.

Anyways, for those of you interested in ’social search’ and search systems, I encourage you to read this paper and tell me your thoughts!

Most research effort does not produce what is thought of as a traditionally publishable result.  That doesn’t mean, however, that nothing was gained by conducting the research.  These results, whether they are failures or merely perplexing, can provide valuable insights into open problems and prevent other researchers from duplicating work.

We have started a journal that focuses on serendipitous (I have no idea why this worked) and unexpected (it seems like this technique should work on this problem but it doesn’t) results.  The goal is to provide a venue for the dissemination and discussion of ideas and to enable more efficient research.

The Journal of Serendipitous and Unexpected Results (JSUR) is an open-access forum for researchers seeking to further scientific discovery by sharing surprising or unexpected results. These results should provide guidance toward the verification (or negation) of extant hypotheses.  JSUR has two branches, one focusing on computational sciences and the other on the life sciences.  JSUR submissions include, but are not limited to, short communications of recent research results, full-length papers, review articles, and opinion pieces.

Recently, we launched the beta version of the journal site at http://jsur.org .  We would love to get your feedback and even better, a submission for the first issue.

To get the journal started, we’re looking to collect a large number of short (2-4 page) reports. I know you have something to publish.

Please help us spread the word and forward this information to interested colleagues.

Looking at the author guidelines, it looks as if submissions will be reviewed both by an editorial board and through a peer review process.  In addition, an interesting note is that all articles will be open-access and Creative-Commons licensed.  I’ll definitely be happy to read some of the articles that come through here once they start publishing.