Based on this post and other comments, it is fairly clear that MG Siegler is on “Team Quora”, calling Quora “a great source of information like Twitter and Facebook and blogs themselves.” Having been a Quora user for several months, I have found the quality of answers on the site to be extremely high. These have ranged from the much-celebrated cases of high-profile individuals answering questions about topics pertaining to them (e.g. Netflix CEO Reed Hastings answering the question “How much does Netflix spend on postage every year?“) to the opportunities for creative individuals to answer questions in awesome and innovative ways (see Wavii programmer Erik Frey‘s answer to the question “Which animal has been used most frequently for a band name?“). I have also personally asked a number of questions and gotten timely and high-quality answers.

However, the key question here is whether the site will continue to be as useful as more and more people join. While many might argue that including more subject matter experts can only improve the site, one must also remember that this increased signal is only useful when it can be separated from the increased noise. Right now, Quora is a bit like <nerd alert>Flynn’s cave dwelling in the Outlands, where the information contained is only safe as long as the masses can’t get to it</nerd alert>; users can trust the answers they find because they often come paired with often famous or at least recognizable names and faces. Yahoo! Answers is a great example of how a Q&A site can decrease in quality with respect to both the questions and answers as it opens up (for a quick, possibly NSFWish laugh, check out “11 Stupid Questions from Yahoo Answers That Have Changed My Life“) Even if you develop the most robust social answer-quality-checker imaginable, the presence of thousands of stupid questions tagged with topic tags that direct them to your inbox is going to turn a lot of the quality answer providers away from the site. In some ways, I felt that Aardvark fell into that trap as it grew more popular, and I now find myself answering a lot of questions that involved identifying rashes.

I had this question about the possible perils of mainstream adoption in mind, when reading a NY Times blog post this week, in which David Pogue describes his first interactions with the site as “a descent into bafflement.” Among the parts of the site that he deems confusing are the login process, the task of adding connections and following topics, and the actual task of asking a question. For chronicling his confusion, however, he earned the following response from Siegler (on his personal blog):

Is this meant to be written from the perspective of a 95-year-old senile man?

Apparently, every site should be designed in a way so that it’s just like every other site that failed before it on the Internet. Makes perfect sense.

Prediction: he’ll love Quora in 12 months.

Now, I would consider David Pogue to be a relatively tech-savvy individual. He’s been blogging about Web/technology topics for over 10 years now, and I generally find his posts to be pretty interesting and insightful. If he is having this much difficulty using the site, I think that his frustrations really do say something about Quora’s current potential to reach beyond the geek crowd into the general public . And on some level, I think that the desire to mock him for this reflects an underlying recognition that the Silicon Valley influentials who currently use the site wouldn’t actually benefit from “regular people” using it, since this could potentially spell the end of Quora’s usefulness. Perhaps a better use of time might be to consider how to make the site more user-friendly and how to maintain the quality as new users arrive.

Studying sites like Wikipedia is a great way to examine how to maintain quality while scaling out to a broader audience. While it’s difficult to ascertain what actually makes Wikipedia work (in practice, if not in theory), many of the qualities inherent in Wikipedia are those identified in a 2005 paper [1] by J.M. Leimeister and colleagues at the Technische Universität München as factors which promote community success in an environment where trusted information is key. Some of these include exposing the identity of content providers, clearly establishing goals for the community, making member profiles available to other members, and providing various levels of anonymity, all of which are things which are built into Wikipedia’s core. Another feature recognized across the literature is recognition and rewards for contributors, something embodied in Wikipedia in the form of “barnstars“. Quora does a great job of exposing identity (which will likely make it much better than Yahoo! Answers), but I don’t believe that it adequately addresses these other elements. I think that the addition of “moderators” or other custodian roles for hyper-motivated users could be the kind of thing that keeps the Quora community in check, and I would be eager to see them roll something like that out before opening the site up to the general public (I think it’s technically still in beta, no?)

Quora is a great source of information for me in some of the same ways that Facebook and Twitter are. The key difference is that in those media, I can control what I see and who it comes from. If you want to imagine the utility of those sites without such controls, just imagine trying to sift through the real-time Twitter stream – it’s mind-numbing, to be kind (unless of course you are a Justin Bieber fan, in which case you should be delighted). The goals of those who currently enjoy Quora’s usefulness should be to help maintain that usefulness as the site grows. I’m not saying that this will be an easy task, but if it isn’t accomplished, I don’t think that anyone is going to love the site in 12 months.

[1] Leimeister, J.M., Ebner, W., & Krcmar, H. 2005: Design, implementation, and evaluation of trust-supporting components in virtual communities for patients. Journal of Management Information Systems 21, 4, 101-135.

Here are some brief notes on the presentation (to avoid the inevitable TL;DR comments):

• Data used was from a study I ran on Mechanical Turk looking at whether the tool WikiDashboard helps people to make different judgments about the credibility of Wikipedia articles.
• Participants placed in 1 of 3 conditions: (WO = Wiki Only, WH = Wiki + the History Page, WD = Wiki + WikiDashboard)
• Articles varied with respect to presumed quality and presumed controversy.
• Using non-parametric tests was fairly straightforward, but none were all that powerful (able to help find interaction effects – one main hope of the study would be to find an interaction between group and quality).

Anyways, this presentation is not supposed to be an expert statistics guide – rather, it represents the results of my research in trying to solve this problem (again, I’m very much not a statistics expert). There are surely many other ways to address the problem, and I would appreciate hearing from others who have tried attacking Likert items for their studies. I am continuing to analyze the data and may post some results in the near future.

Eugene Eric Kim: Strategy Program Manager

• If you include all of the component sites, Wikimedia is the 5th most accessed web-property in the world.
• 350M regular visitors, $10M in revenue, and only 35 employees.
• 45K active contributors (a term they use to indicate people who make 5 or more edits per month) on English Wikipedia.
• The country with the most visitors is actually Canada (which nobody in the audience guessed).
• Defined the Wikimedia Foundation mission with a Jimmy Wales quote: “Imagine a world in which every single person on the planet is given free access to the sum of all human knowledge.“

Trevor Parscal: Lead Front-End, UX Programs

• Trevor is the guy in charge of “basically everything you see” (wow!)
• Wikimedia research shows that people don’t find the software easy to use (duh), so they have launched the Usability Initiative.
• In fact, when they were testing with users, they had one user who took 20 minutes to figure out how to edit a page (and this wasn’t entirely out of the ordinary).
• Asking people what they wanted in the site proved not-so-successful, but having them try out a new Beta version and observing behavior was really fruitful.
• As of now, 84% of the people who opted into the Wikipedia Beta have stayed (almost 300K) people – (there was no mention of how to find the beta, btw).

Tomasz Finc: Engineering Program Manager

• Fundraising is done annually, between November and January.
• Amount raised: 2006 – $1m, 2007 – $2M, 2008 – $6M, and 2009 – $8.1M
• Most of their fundraising comes from small donations (contrary to usual trend of large donations for these types of efforts)
• Did a lot of A/B style testing to figure out how to optimize contribution – a lot of this is actually shared on the Wikimedia Blog.
• Adding Jimmy Wales’ plea increased the donations a LOT (so much that at first they thought the site was being attacked).
• The iPhone application and mobile gateway are both being developed by the community.
• The OLPC now has a full copy of the English Wikipedia on it.

As you can see, the talks basically focused on three elements: 1) Wikipedia is big and wants to get bigger, 2) Wikipedia is hard to use and wants to get easier, 3) Wikipedia relies a lot on the community. While there wasn’t much that was earth-shattering, each of these elements was pretty interesting – the idea that such a HUGE platform and vast amount of content can be supported by just 35 full-time employees and the contributions of the community is incredible, and speaks to the power that effective community management can bring. As Wikipedia is one of the greatest examples of social software and content production, it was great to get the opportunity to peer under the hood a little bit.

For some more information that may not have made it into this brain-dump, check out my live-tweet of the event here.