Here are some brief notes on the presentation (to avoid the inevitable TL;DR comments):

• Data used was from a study I ran on Mechanical Turk looking at whether the tool WikiDashboard helps people to make different judgments about the credibility of Wikipedia articles.
• Participants placed in 1 of 3 conditions: (WO = Wiki Only, WH = Wiki + the History Page, WD = Wiki + WikiDashboard)
• Articles varied with respect to presumed quality and presumed controversy.
• Using non-parametric tests was fairly straightforward, but none were all that powerful (able to help find interaction effects – one main hope of the study would be to find an interaction between group and quality).

Anyways, this presentation is not supposed to be an expert statistics guide – rather, it represents the results of my research in trying to solve this problem (again, I’m very much not a statistics expert). There are surely many other ways to address the problem, and I would appreciate hearing from others who have tried attacking Likert items for their studies. I am continuing to analyze the data and may post some results in the near future.

Poisson distribution. Poisson process. Poisson equation. Poisson kernel. Poisson regression. Poisson summation formula. Poisson’s spot. Poisson’s ratio. Poisson bracket. Euler-Poisson-Darboux equation. This is only a partial list, and yet it shows the degree to which Siméon-Denis Poisson’s work has impacted just about all branches of science. But what is so impressive is not the volume of his contributions but rather their depth, raising a puzzling question: How did Poisson manage to work simultaneously on so many quite different problems and yet stay sufficiently focused to offer deep and lasting contributions?

Well, we had a secret: a notebook and a tiny habit.

Each time Poisson encountered a problem he though fascinating, he would resist the temptation to savor it. He pulled out his notebook instead and made a note of it and promptly returned to the problem that had absorbed him before the interruption. Once he solved the problem at hand, he mulled over the list of problems scribbled in his notebook, then picking as his next challenge the one he found the most interesting.

Poisson’s little secret was lifelong, careful prioritizing.

So, this essentially describes the polar opposite of my work habits, which currently consist of frenetically switching from task to task to ensure that I complete none of them. I’m thinking of giving the priority list a try – has anybody tried a scheme like this and had success with it? Would be curious to hear your story!

Eugene Eric Kim: Strategy Program Manager

• If you include all of the component sites, Wikimedia is the 5th most accessed web-property in the world.
• 350M regular visitors, $10M in revenue, and only 35 employees.
• 45K active contributors (a term they use to indicate people who make 5 or more edits per month) on English Wikipedia.
• The country with the most visitors is actually Canada (which nobody in the audience guessed).
• Defined the Wikimedia Foundation mission with a Jimmy Wales quote: “Imagine a world in which every single person on the planet is given free access to the sum of all human knowledge.“

Trevor Parscal: Lead Front-End, UX Programs

• Trevor is the guy in charge of “basically everything you see” (wow!)
• Wikimedia research shows that people don’t find the software easy to use (duh), so they have launched the Usability Initiative.
• In fact, when they were testing with users, they had one user who took 20 minutes to figure out how to edit a page (and this wasn’t entirely out of the ordinary).
• Asking people what they wanted in the site proved not-so-successful, but having them try out a new Beta version and observing behavior was really fruitful.
• As of now, 84% of the people who opted into the Wikipedia Beta have stayed (almost 300K) people – (there was no mention of how to find the beta, btw).

Tomasz Finc: Engineering Program Manager

• Fundraising is done annually, between November and January.
• Amount raised: 2006 – $1m, 2007 – $2M, 2008 – $6M, and 2009 – $8.1M
• Most of their fundraising comes from small donations (contrary to usual trend of large donations for these types of efforts)
• Did a lot of A/B style testing to figure out how to optimize contribution – a lot of this is actually shared on the Wikimedia Blog.
• Adding Jimmy Wales’ plea increased the donations a LOT (so much that at first they thought the site was being attacked).
• The iPhone application and mobile gateway are both being developed by the community.
• The OLPC now has a full copy of the English Wikipedia on it.

As you can see, the talks basically focused on three elements: 1) Wikipedia is big and wants to get bigger, 2) Wikipedia is hard to use and wants to get easier, 3) Wikipedia relies a lot on the community. While there wasn’t much that was earth-shattering, each of these elements was pretty interesting – the idea that such a HUGE platform and vast amount of content can be supported by just 35 full-time employees and the contributions of the community is incredible, and speaks to the power that effective community management can bring. As Wikipedia is one of the greatest examples of social software and content production, it was great to get the opportunity to peer under the hood a little bit.

For some more information that may not have made it into this brain-dump, check out my live-tweet of the event here.

• Identifying the features that underlie “tag quality” (e.g. Sen, et al. (2007), Zhang, et al. (2009))
• Topic-based approaches for information retrieval from tagged collections (e.g. Zhou, et al. (2008))
• Graph-based algorithms for ranking based on user tags (e.g. Hotho, et al. (2006), Noll, et al. (2009))

I decided to try a little Mechanical Turk study to see if I could spot some differences between tags generated by experts and those generated by novices. I had each Turker read 1 of 5 web pages (on the topic of “enterprise 2.0 mashups”) and enter 5 tags which they thought would be useful for bookmarking the page (either for themselves or others). I also asked them to rate how familiar they were with the subject matter (“Not at All”, “Slightly Familiar”, “Somewhat Familiar”, and “I am an Expert”).

As a game, I thought it would be interesting to post some of the responses to see how easy it was to identify which tags were generated by people who rated themselves as “experts” vs. “non-experts”. I took all of the tags generated by each expertise group, cleaned them up for minor spelling mistakes and typos (e.g., “applciation” > “application”) and generated a tag cloud using Wordle, where the tag size corresponds to the frequency of use of that word (all other factors, such as positioning and color, are purely stylistic).

For the following URL – http://www.soamag.com/I18/0508-1.php – can you identify which tag cloud belongs to which of these groups: “Not at All (Familiar)”, “Slightly Familiar”, and “Somewhat Familiar” (there was a 4th category of “I am an Expert”, but nobody rating this URL classified themselves this way):

Tag Cloud 1 (N = 17)

Tag Cloud 2 (N = 16)

Tag Cloud 3 (N = 14)

If you have any idea which tag cloud is which, please feel free to post your guess in the comments! I’d be extremely curious to see why people guessed the way that they did. I am actually currently in the process of having some Turkers do the same thing; if you are curious about the answers, come back for my follow-up post where I post the correct answers, as well as the results of the Mechanical Turk evaluation of the tag cloud.

THE OBJECT OF THE AUTHOR in the following pages has been to collect the most remarkable instances of those moral epidemics which have been excited, sometimes by one cause and sometimes by another, and to show how easily the masses have been led astray, and how imitative and gregarious men are, even in their infatuations and crimes.

Charles Mackay – Memoirs of Extraordinary Popular Delusions and the Madness of the Crowds

Image courtesy of Marcos Vasconcelos (click for original)

I’ve been interested a bit in looking at how to help people find high-quality information on the web – recently, I have been exploring how to help people make better credibility judgments about the information they find. One paper I was reading, “Statement Map: Assisting Information Credibility Analysis by Visualizing Arguments” by Koji Murakami and others at the Nara Institute of Science and Technology in Japan, uses as a motivating example the recent movement against vaccinations for children, specifically the MMR (Measles, Mumps, and Rubella), as the result of fears that these vaccines could cause autism.

Back in 2003, Pew reported that over 80% of Internet users have searched for health information (such as info about fitness or vaccinations) online, and one can imagine that this number has only grown since then, so this example is important in illustrating the potential impact of such health memes. I’ve heard multiple parents mention the supposed vaccination-autism link before when making decisions about whether or not to vaccinate their children, so it’s extremely important to figure out how to make sure these parents get intelligent, credible information when searching on the web. The Murakami paper provides some interesting background information on how this particular meme first started:

In 1997, a group of researchers in the UK lead (sic) by Dr. Andrew Wakefield published a study implying a causal connection between Measles, Mumps, and Rubella (MMR) vaccinations and the development of autism in children. Though further scrutiny of these initial results  disproved the autism-vaccination link – culminating in the withdrawal of endorsements by 10 of the study’s 12 authors – the damage had already been done.

The consequences of this single, spurious, study have already been far-reaching. The resulting backlash precipitated a drop in vaccination rates in the UK (where the study was first published), which has led to an increase in outbreaks of measles over the past decade to the point where measles are once again (after transmission was halted 14 years ago) being considered endemic. Even vaccination rates here in the Bay Area have dropped, with the Examiner reporting that vaccination rates are as low as 50% for some Bay Area schools.

I suppose a lot of this can be chalked up to mainstream media coverage of the original study (as well as coverage of well-meaning, but misguided celebrity activists like Jenny McCarthy). However, a large part of why this meme has continued even after the original study was shown to be dubious is due to social phenomena such as communal reinforcement (or the “millions of people can’t be wrong” phenomenon) that can occur so easily on the web. Because it is so easy to publish information on the web, and because information published tends to persist, it is easy to find a wealth of documents supporting any viewpoint, no matter how much evidence there actually is to support that claim. In this case, one can read 100 different news articles, blog posts, and other online resources based on the Wakefield, et al. study without knowing first that these stories do not corroborate each other (as they are drawn from the same small, possibly falsified study) and second that the original study was actually recently retracted by the journal in the first place.

The way that these stories snowball and take on a life of their own is something that Charles Mackay documented in his book “Extraordinary Popular Delusions and the Madness of the Crowds” (the first lines of which are quoted above). He tackles a variety of subjects ranging from the Dutch tulip craze of the 16th century (cf. “21st century housing bubble“) to alchemy to witch hunts. From this book (written in 1841), we can see that the often reasonable shortcuts that people make when processing new information can sometimes lead to these self-propagating effects which take on a life of their own.

The unfortunate fact is that just because the web gives us access to more information doesn’t guarantee that we are going to choose and use it wisely. This is why building tools to help people make better credibility judgments online is so important, raising two questions:

• How do we extract data from within a single web page to help people make better judgments about the information it contains? I know that there is a good deal of work on this topic in Wikipedia with tools like Wikipedia Scanner and PARC’s WikiDashboard helping to to expose author and change information, but how can we bring tools like these to the web as a whole?
• How do we connect data across web pages to hep propagate changes in information across the web? As an example, if information about the study’s retraction could be propagated to pages reporting on the study, parents reading those pages would be less likely to be led astray, possibly saving lives.

For those who are specifically interested in the MMR vaccine controversy, the Wikipedia page links to a lot of good resources, including a long list of studies conducted in the last decade which show no link between autism and the vaccine.

Given Google’s recent social efforts (Twitter Search, Social Search, Google Buzz, etc.), I am curious to see what they will do with the Aardvark product – will it stand alone as it has or will it find its way into existing or new Google tools? I, for one, would love to see it integrated into Google’s main search. One consequence of Google’s recent launch of Buzz is reminding people that Google has been collecting data on your social network for a while now. If Aardvark were integrated into your Google network, we’d have a out-of-the-box solution for social search (no messy profile-connecting or friend-inviting needed! To me, it seems like one of the biggest hurdles for most people in terms of social search or networking tools is the cost of building up their networks, so this would provide a quick and easy way around that.

What will GoogVark look like? I can’t say I’ve ever used the “I’m Feeling Lucky” button, but I do know that there are times when I can’t quite find the best answers through Google search, and I’d love to be able to seamlessly shift over to social search. I personally would love to see something like this (with an example supplied by Google itself!):

Would this make social search more inviting to you?

P.S. Congratulations to Aardvark’s founders over at The Mechanical Zoo – you guys deserve it!

Most research effort does not produce what is thought of as a traditionally publishable result.  That doesn’t mean, however, that nothing was gained by conducting the research.  These results, whether they are failures or merely perplexing, can provide valuable insights into open problems and prevent other researchers from duplicating work.

We have started a journal that focuses on serendipitous (I have no idea why this worked) and unexpected (it seems like this technique should work on this problem but it doesn’t) results.  The goal is to provide a venue for the dissemination and discussion of ideas and to enable more efficient research.

The Journal of Serendipitous and Unexpected Results (JSUR) is an open-access forum for researchers seeking to further scientific discovery by sharing surprising or unexpected results. These results should provide guidance toward the verification (or negation) of extant hypotheses.  JSUR has two branches, one focusing on computational sciences and the other on the life sciences.  JSUR submissions include, but are not limited to, short communications of recent research results, full-length papers, review articles, and opinion pieces.

Recently, we launched the beta version of the journal site at http://jsur.org .  We would love to get your feedback and even better, a submission for the first issue.

To get the journal started, we’re looking to collect a large number of short (2-4 page) reports. I know you have something to publish.

Please help us spread the word and forward this information to interested colleagues.

Looking at the author guidelines, it looks as if submissions will be reviewed both by an editorial board and through a peer review process.  In addition, an interesting note is that all articles will be open-access and Creative-Commons licensed.  I’ll definitely be happy to read some of the articles that come through here once they start publishing.

Upon reading this study, I noticed two things:

1) The original experiment solicited participants by offering to donate $10 to charity for their time; resulting in a net cost of at least $14K, raising the second question: Given the growth of sites like Mechanical Turk, can we get comparable results for less money?

2) The data was originally collected in December 2009 1999, which is almost exactly 10 years ago. Content on the web and our interactions with it have changed a great deal since then – I mean, in 1999, Google had only existed for a year, blogs were still relatively uncommon (LiveJournal had just started), and Wikipedia didn’t even exist yet! This raised the question: Given that the average Internet user now has a far greater amount of experience navigating the Web, should we expect the responses to be different 10 years later?

I decided to explore both of these questions through a survey on Mechanical Turk; in December 2009, I posted a HIT to Mechanical Turk replicating the original study as closely as possible. The 51 statements about web site elements were available from Fogg’s original paper, the same 7-point Likert scale was used, and the survey items were randomized.  I paid $0.05 per HIT, and I ended up getting 327 responses, with none thrown out due to quality.

An initial, high-level examination of the responses showed that they actually matched the 1999 data fairly well.  Average Likert ratings for each item correlated highly with average ratings for the same items in the 1999 data with R^2 = 0.96.  One difference in the data was that answers were compressed (closer to 0) overall, so for the purposes of comparison, I transformed the 2009 data using the transformation (1.1677X – 0.0003) to match the 1999 data (note that this has no effect on the correlation coefficient).

The first analysis in Fogg, et al. was to separate the various elements into 7 “scales” using factor analysis (Real-World Feel, Ease of Use, Expertise, Trustworthiness, Tailoring, Commercial Implications, and Amateurism).  Below, I present comparisons for items in each scale.  I highlighted items that deviated from the 1999 values by more than 0.25 (without the original data, I couldn’t do much more in-depth comparison), but this might give some rough idea of which elements are more important now than they were in 1999 (red) and which elements are less important (purple):

Elements related to the "Real-World" feel scale are rated similarly from 1999 to 2009.

One difference is that people seem more critical of long download times than in the past.

For some reason, it seems that displaying an award helps your site's credibility more than 10 years ago, and that providing a lot of news stories matters less.

Stating your policy on content and ending in ".org" are more important to people now - could this be a cultural shift in response to sites like Wikipedia? Overall, it seems as if links out to other sites matter less for credibility now.

Email confirmations are more important now than in 1999.

Outside advertisements and an e-commerce focus matter more for credibility now than in the past. People are paying less attention to the commercial purpose of sites, as well as the number of ads and their integration with content (Google is a favorite site for many, after all.)

People seem to notice domain name mismatches now (more public knowledge of phishing/identity theft now?), but less attention to multi-lingual sites.

People seem more willing to trust free financial sites now than in the past.

Anyways, as we might have guessed, the answers from 2009 seem to match the answers from 1999 pretty well.  The elements that made things highly credible or highly ‘un-credible’ in the past seem to have remained constant, and those which didn’t matter then seem not to matter too much now.  Some interesting elements are noted in the captions with some rough conjectures as to why some of them might be trending the way they are.

The rest of the Fogg paper focused on characterizing differences in scale responses due to demographic differences between participants, but I found that part of the study less convincing as they averaged over the positively and negatively phrased elements on each scale (which I think makes the interpretation somewhat confusing).

Now, back to our two questions:

1) It looks as if using Mechanical Turk, we were able to get reasonable answers that fairly closely matched those from the original study.  Total price tag: 327 response * $0.05 = $16.35 paid to participants (plus Amazon’s cut), which made this study about $13,980 cheaper than the original one.

2) We see above a few small changes in what makes web sites credible, but overall, people are looking at the same things, meaning that we should continue taking the same factors into account when designing websites. I’ve made some guesses as to why these elements may have changed over time, but I’m curious to hear what you think, so leave a comment!

——————————————————-

Last month, PARC hosted the first of two National Science Foundation-funded workshops on Technology-Mediated Social Participation, co-organized by PARC’s Peter Pirolli (see his original post on why we were motivated to do this here) and the University of Maryland’s Jennifer Preece and Ben Shneiderman. With the goal of drawing up a strong scientific research agenda and educational recommendations to help foster a new era of technologies that support social participation, the workshop brought together some of the top U.S. industry and academia researchers from fields such as Computer Science, Information Science, and Cognitive Science.

Workshop themes

Specifically, the workshop addressed three major themes:

1. How to integrate theory across levels from the individual to the community.
2. How to develop new methods of measuring social connections and social capital across networks.
3. How to build an infrastructure to collect data reliably and responsibly.

The primary goal of the workshop has been to produce reports on these three topics — now available on the public TMSP wiki. The reports address participants’ government, academic, and education recommendations, and implications for each of the topics.

Motivations and timing

Amy Bruckman of Georgia Tech reminded us of the need for such a cross-cutting initiative, commenting that “Right now it’s harder than it should be for academia and industry to work together. I think that’s a key place where a government-funded initiative can help — making it easier to form industry-academia collaborations.” Ben Shneiderman enthused that the workshop was a “remarkable gathering of leading academic and industry researchers and that it served as a big step forward in realizing the goals set out in my letter in AAAS Science (13 March 2009).”

Thinking Big! (Photo originally posted by @edchi)

Preece (serving as the Principal Investigator for the NSF grant supporting the workshop) spoke to the timeliness of this effort, observing that “A few people have been investigating this topic since the 1990’s, but now that billions of people worldwide are using Web 2.0 social media, such as Facebook, wikis, blogs, microblogs, and discussion forums, substantially increased research is needed to understand the benefits and dangers.” Throughout the workshop, one theme that emerged was how a more thorough understanding of social participation could affect users at all levels of participation in a variety of domains; Preece noted, “If community managers knew how to increase participation, then this technology could be more successful for patient support, citizen science contributions, and changing behaviors for energy conservation.”

PARC Forum and audience participation

The workshop also featured a special PARC Forum panel discussion on the Future of Technology-Mediated Social Participation, which discussed patterns of attention in social media, the potential of peer-produced content, and leveraging social network data to model and understand behavior. Bruckman (who served as a panelist along with Shneiderman, HP Labs’ Bernardo Huberman, and Facebook’s Cameron Marlow) shared, “Aside from joking about avocados and bison…I do think the panel opened some questions that fed into the discussion we had the next day,” highlighting the impact that public participation can have on driving this initiative.

Cameron Marlow discussing the relative dangers of avocados and bread (Photo originally posted by @HCIR_GeneG)

How can you get involved?

For those interested in helping to support this initiative — or curious about technology and social systems — the final reports will provide a glimpse into the ongoing discussion. In addition, the TMSP website also features a growing list of relevant resources, such as books, courses, and research groups (compiled, in part, by U. Maryland graduate student Dana Rotman, who has been instrumental in the planning and execution of these workshops).

What’s next?

Shneiderman hopes the effort will “catalyze the research community and encourage the U.S. National Science Foundation to dramatically increase its support for Technology-Mediated Social Participation, which is aligned with national priorities such as healthcare, energy conservation, disaster response, community safety, and more.”

The next event, the East Coast Workshop, will take place 11-12 February in Virginia (more information will be available at the TMSP East Coast Workshop website.)

And, we welcome you to read the reports, watch the Forum, and add your insights!

——————————————————-

Back to my personal comments…

All in all, I am extremely appreciative of the opportunity to participate in the workshop. I met a lot of incredible people, and I got to participate in some fascinating discussion. A lot of innovative ideas were discussed, and I am eager to see what will eventually emerge from these workshops – be it new research centers, changes in governmental policy, or increased funding for social computing research.

STATS: He started off with some data about internet usage (many pulled from here), most notably mentioning that there are currently approximately 1.7B Internet users in the world (also adding that as Google’s ‘Internet Evangelist’, he “still has 75% of the world left to go.”).  What was interesting was his focus on what the numbers really meant – while Asia only has an 18.5% Internet penetration rate, that still works out to about 704M people, which is still more than 2x the entire US population.  In addition, he mentioned that there are currently 4 billion mobile phones, a fact which was new to me, and which implied that most people were using mobile phones as their primary conduit to the Internet.

Major Changes: He also shared some of the major changes that were happening soon.  One of these was about the Internet’s current switch to IPv6, mentioning that we were on track to exhaust the stock of IPv4 addresses by 2011 (perhaps sooner if there is a rush for addresses at the end).  He mentioned that the new 128-bit addresses would allow for 3.8 x 10^34 addresses – “a number only Congress can appreciate”.  In addition, he mentioned upcoming changes like internationalization of domain names (allowing non-latin characters in top-level domain names).

Applications: He then did a whirlwind tour of the kinds of applications that are supported by the Internet (mostly seen through Google’s eyes, of course).  Email (GMAIL), Video-Sharing (YOUTUBE), Maps (GMAPS), you get the picture…It was interesting to hear him talk about Google Wave, because even though he spoke about it with conviction, I noticed that it was still difficult for him to really express the use cases for the service, something that I (and many other people) have had a bit of trouble with.

New Types of Devices: He talked about the grand proliferation of internet-enabled things.  This is a topic that I’ve long had some interest in and have been desirous to get involved with (beyond playing around with my Nabaztag, of course).  He actually got into an interesting anecdote about installing temperature sensors in his wine cellar that text messaged him when the temperature rose above 60 degrees.  He talked about possible additions to this project, including actuators to turn on the cooling system remotely, and RFID-indexing all of the bottles to keep inventory.  This transitioned into discussion of putting sensors in the corks themselves to deliver information about the wine, which reminded me a great deal of Bruce Sterling’s discussion of “spimes” in his book, “Shaping Things” (a quick, interesting read, btw – definitely recommended).

Challenges of the Digital Age: Finally (skipping over some things), he talked about 2 big challenges that he saw for information handling in the future.  The first was the phenomenon of ‘Bit Rot’ – he applied the term not to the decay of physical storage media, but rather to the idea that we might just stop deciding to update programs, and thus 1000 years from now, we’ll never be able to see that Powerpoint presentation from 2004.  The other grand challenge he brought up (something he has been heavily involved with) was the creation of the “Interplanetary Internet“.  The idea (linking things in space to the Internet) is relatively simple, but the execution (as one might expect) is somewhat hard.  Challenges such as vast distances (speed of light actually becomes a factor when Mars gets as far as 235 million miles from Earth) and planetary rotation (now you see it, now you don’t), have led them to develop a Delay-Tolerant Network protocol that uses a “store and forward” approach instead of trying to achieve end-to-end communication.  Their plan is to use as nodes satellites which have been re-purposed and re-programmed after completing their original mission.

Anyways, the talk, which covered everything from mobile phones to cloud-computing to internet-enabled surfboards to Googling in space, was interesting and inspiring.  In case you want to watch it, PARC has started putting up videos of the PARC Forum talks, and you can find the Vint Cerf talk here a few days hence.