Based on this post and other comments, it is fairly clear that MG Siegler is on “Team Quora”, calling Quora “a great source of information like Twitter and Facebook and blogs themselves.” Having been a Quora user for several months, I have found the quality of answers on the site to be extremely high. These have ranged from the much-celebrated cases of high-profile individuals answering questions about topics pertaining to them (e.g. Netflix CEO Reed Hastings answering the question “How much does Netflix spend on postage every year?“) to the opportunities for creative individuals to answer questions in awesome and innovative ways (see Wavii programmer Erik Frey‘s answer to the question “Which animal has been used most frequently for a band name?“). I have also personally asked a number of questions and gotten timely and high-quality answers.

However, the key question here is whether the site will continue to be as useful as more and more people join. While many might argue that including more subject matter experts can only improve the site, one must also remember that this increased signal is only useful when it can be separated from the increased noise. Right now, Quora is a bit like <nerd alert>Flynn’s cave dwelling in the Outlands, where the information contained is only safe as long as the masses can’t get to it</nerd alert>; users can trust the answers they find because they often come paired with often famous or at least recognizable names and faces. Yahoo! Answers is a great example of how a Q&A site can decrease in quality with respect to both the questions and answers as it opens up (for a quick, possibly NSFWish laugh, check out “11 Stupid Questions from Yahoo Answers That Have Changed My Life“) Even if you develop the most robust social answer-quality-checker imaginable, the presence of thousands of stupid questions tagged with topic tags that direct them to your inbox is going to turn a lot of the quality answer providers away from the site. In some ways, I felt that Aardvark fell into that trap as it grew more popular, and I now find myself answering a lot of questions that involved identifying rashes.

I had this question about the possible perils of mainstream adoption in mind, when reading a NY Times blog post this week, in which David Pogue describes his first interactions with the site as “a descent into bafflement.” Among the parts of the site that he deems confusing are the login process, the task of adding connections and following topics, and the actual task of asking a question. For chronicling his confusion, however, he earned the following response from Siegler (on his personal blog):

Is this meant to be written from the perspective of a 95-year-old senile man?

Apparently, every site should be designed in a way so that it’s just like every other site that failed before it on the Internet. Makes perfect sense.

Prediction: he’ll love Quora in 12 months.

Now, I would consider David Pogue to be a relatively tech-savvy individual. He’s been blogging about Web/technology topics for over 10 years now, and I generally find his posts to be pretty interesting and insightful. If he is having this much difficulty using the site, I think that his frustrations really do say something about Quora’s current potential to reach beyond the geek crowd into the general public . And on some level, I think that the desire to mock him for this reflects an underlying recognition that the Silicon Valley influentials who currently use the site wouldn’t actually benefit from “regular people” using it, since this could potentially spell the end of Quora’s usefulness. Perhaps a better use of time might be to consider how to make the site more user-friendly and how to maintain the quality as new users arrive.

Studying sites like Wikipedia is a great way to examine how to maintain quality while scaling out to a broader audience. While it’s difficult to ascertain what actually makes Wikipedia work (in practice, if not in theory), many of the qualities inherent in Wikipedia are those identified in a 2005 paper [1] by J.M. Leimeister and colleagues at the Technische Universität München as factors which promote community success in an environment where trusted information is key. Some of these include exposing the identity of content providers, clearly establishing goals for the community, making member profiles available to other members, and providing various levels of anonymity, all of which are things which are built into Wikipedia’s core. Another feature recognized across the literature is recognition and rewards for contributors, something embodied in Wikipedia in the form of “barnstars“. Quora does a great job of exposing identity (which will likely make it much better than Yahoo! Answers), but I don’t believe that it adequately addresses these other elements. I think that the addition of “moderators” or other custodian roles for hyper-motivated users could be the kind of thing that keeps the Quora community in check, and I would be eager to see them roll something like that out before opening the site up to the general public (I think it’s technically still in beta, no?)

Quora is a great source of information for me in some of the same ways that Facebook and Twitter are. The key difference is that in those media, I can control what I see and who it comes from. If you want to imagine the utility of those sites without such controls, just imagine trying to sift through the real-time Twitter stream – it’s mind-numbing, to be kind (unless of course you are a Justin Bieber fan, in which case you should be delighted). The goals of those who currently enjoy Quora’s usefulness should be to help maintain that usefulness as the site grows. I’m not saying that this will be an easy task, but if it isn’t accomplished, I don’t think that anyone is going to love the site in 12 months.

[1] Leimeister, J.M., Ebner, W., & Krcmar, H. 2005: Design, implementation, and evaluation of trust-supporting components in virtual communities for patients. Journal of Management Information Systems 21, 4, 101-135.

Here are some brief notes on the presentation (to avoid the inevitable TL;DR comments):

• Data used was from a study I ran on Mechanical Turk looking at whether the tool WikiDashboard helps people to make different judgments about the credibility of Wikipedia articles.
• Participants placed in 1 of 3 conditions: (WO = Wiki Only, WH = Wiki + the History Page, WD = Wiki + WikiDashboard)
• Articles varied with respect to presumed quality and presumed controversy.
• Using non-parametric tests was fairly straightforward, but none were all that powerful (able to help find interaction effects – one main hope of the study would be to find an interaction between group and quality).

Anyways, this presentation is not supposed to be an expert statistics guide – rather, it represents the results of my research in trying to solve this problem (again, I’m very much not a statistics expert). There are surely many other ways to address the problem, and I would appreciate hearing from others who have tried attacking Likert items for their studies. I am continuing to analyze the data and may post some results in the near future.

Upon reading this study, I noticed two things:

1) The original experiment solicited participants by offering to donate $10 to charity for their time; resulting in a net cost of at least $14K, raising the second question: Given the growth of sites like Mechanical Turk, can we get comparable results for less money?

2) The data was originally collected in December 2009 1999, which is almost exactly 10 years ago. Content on the web and our interactions with it have changed a great deal since then – I mean, in 1999, Google had only existed for a year, blogs were still relatively uncommon (LiveJournal had just started), and Wikipedia didn’t even exist yet! This raised the question: Given that the average Internet user now has a far greater amount of experience navigating the Web, should we expect the responses to be different 10 years later?

I decided to explore both of these questions through a survey on Mechanical Turk; in December 2009, I posted a HIT to Mechanical Turk replicating the original study as closely as possible. The 51 statements about web site elements were available from Fogg’s original paper, the same 7-point Likert scale was used, and the survey items were randomized.  I paid $0.05 per HIT, and I ended up getting 327 responses, with none thrown out due to quality.

An initial, high-level examination of the responses showed that they actually matched the 1999 data fairly well.  Average Likert ratings for each item correlated highly with average ratings for the same items in the 1999 data with R^2 = 0.96.  One difference in the data was that answers were compressed (closer to 0) overall, so for the purposes of comparison, I transformed the 2009 data using the transformation (1.1677X – 0.0003) to match the 1999 data (note that this has no effect on the correlation coefficient).

The first analysis in Fogg, et al. was to separate the various elements into 7 “scales” using factor analysis (Real-World Feel, Ease of Use, Expertise, Trustworthiness, Tailoring, Commercial Implications, and Amateurism).  Below, I present comparisons for items in each scale.  I highlighted items that deviated from the 1999 values by more than 0.25 (without the original data, I couldn’t do much more in-depth comparison), but this might give some rough idea of which elements are more important now than they were in 1999 (red) and which elements are less important (purple):

Elements related to the "Real-World" feel scale are rated similarly from 1999 to 2009.

One difference is that people seem more critical of long download times than in the past.

For some reason, it seems that displaying an award helps your site's credibility more than 10 years ago, and that providing a lot of news stories matters less.

Stating your policy on content and ending in ".org" are more important to people now - could this be a cultural shift in response to sites like Wikipedia? Overall, it seems as if links out to other sites matter less for credibility now.

Email confirmations are more important now than in 1999.

Outside advertisements and an e-commerce focus matter more for credibility now than in the past. People are paying less attention to the commercial purpose of sites, as well as the number of ads and their integration with content (Google is a favorite site for many, after all.)

People seem to notice domain name mismatches now (more public knowledge of phishing/identity theft now?), but less attention to multi-lingual sites.

People seem more willing to trust free financial sites now than in the past.

Anyways, as we might have guessed, the answers from 2009 seem to match the answers from 1999 pretty well.  The elements that made things highly credible or highly ‘un-credible’ in the past seem to have remained constant, and those which didn’t matter then seem not to matter too much now.  Some interesting elements are noted in the captions with some rough conjectures as to why some of them might be trending the way they are.

The rest of the Fogg paper focused on characterizing differences in scale responses due to demographic differences between participants, but I found that part of the study less convincing as they averaged over the positively and negatively phrased elements on each scale (which I think makes the interpretation somewhat confusing).

Now, back to our two questions:

1) It looks as if using Mechanical Turk, we were able to get reasonable answers that fairly closely matched those from the original study.  Total price tag: 327 response * $0.05 = $16.35 paid to participants (plus Amazon’s cut), which made this study about $13,980 cheaper than the original one.

2) We see above a few small changes in what makes web sites credible, but overall, people are looking at the same things, meaning that we should continue taking the same factors into account when designing websites. I’ve made some guesses as to why these elements may have changed over time, but I’m curious to hear what you think, so leave a comment!