The presentation ends with a quick intro in the JavaScript code (from the Iterative Text Improvement example on the TurKit website) and some hopefully helpful information to know when using the Java application that you can download to try out TurKit. If I survive the end of the quarter, I hope to get a post up with some TurKit tutorial tips and lessons learned. If you have questions about TurKit, let me know, and I’ll try to get them answered in the next post!

• Identifying the features that underlie “tag quality” (e.g. Sen, et al. (2007), Zhang, et al. (2009))
• Topic-based approaches for information retrieval from tagged collections (e.g. Zhou, et al. (2008))
• Graph-based algorithms for ranking based on user tags (e.g. Hotho, et al. (2006), Noll, et al. (2009))

I decided to try a little Mechanical Turk study to see if I could spot some differences between tags generated by experts and those generated by novices. I had each Turker read 1 of 5 web pages (on the topic of “enterprise 2.0 mashups”) and enter 5 tags which they thought would be useful for bookmarking the page (either for themselves or others). I also asked them to rate how familiar they were with the subject matter (“Not at All”, “Slightly Familiar”, “Somewhat Familiar”, and “I am an Expert”).

As a game, I thought it would be interesting to post some of the responses to see how easy it was to identify which tags were generated by people who rated themselves as “experts” vs. “non-experts”. I took all of the tags generated by each expertise group, cleaned them up for minor spelling mistakes and typos (e.g., “applciation” > “application”) and generated a tag cloud using Wordle, where the tag size corresponds to the frequency of use of that word (all other factors, such as positioning and color, are purely stylistic).

For the following URL – http://www.soamag.com/I18/0508-1.php – can you identify which tag cloud belongs to which of these groups: “Not at All (Familiar)”, “Slightly Familiar”, and “Somewhat Familiar” (there was a 4th category of “I am an Expert”, but nobody rating this URL classified themselves this way):

Tag Cloud 1 (N = 17)

Tag Cloud 2 (N = 16)

Tag Cloud 3 (N = 14)

If you have any idea which tag cloud is which, please feel free to post your guess in the comments! I’d be extremely curious to see why people guessed the way that they did. I am actually currently in the process of having some Turkers do the same thing; if you are curious about the answers, come back for my follow-up post where I post the correct answers, as well as the results of the Mechanical Turk evaluation of the tag cloud.

Upon reading this study, I noticed two things:

1) The original experiment solicited participants by offering to donate $10 to charity for their time; resulting in a net cost of at least $14K, raising the second question: Given the growth of sites like Mechanical Turk, can we get comparable results for less money?

2) The data was originally collected in December 2009 1999, which is almost exactly 10 years ago. Content on the web and our interactions with it have changed a great deal since then – I mean, in 1999, Google had only existed for a year, blogs were still relatively uncommon (LiveJournal had just started), and Wikipedia didn’t even exist yet! This raised the question: Given that the average Internet user now has a far greater amount of experience navigating the Web, should we expect the responses to be different 10 years later?

I decided to explore both of these questions through a survey on Mechanical Turk; in December 2009, I posted a HIT to Mechanical Turk replicating the original study as closely as possible. The 51 statements about web site elements were available from Fogg’s original paper, the same 7-point Likert scale was used, and the survey items were randomized.  I paid $0.05 per HIT, and I ended up getting 327 responses, with none thrown out due to quality.

An initial, high-level examination of the responses showed that they actually matched the 1999 data fairly well.  Average Likert ratings for each item correlated highly with average ratings for the same items in the 1999 data with R^2 = 0.96.  One difference in the data was that answers were compressed (closer to 0) overall, so for the purposes of comparison, I transformed the 2009 data using the transformation (1.1677X – 0.0003) to match the 1999 data (note that this has no effect on the correlation coefficient).

The first analysis in Fogg, et al. was to separate the various elements into 7 “scales” using factor analysis (Real-World Feel, Ease of Use, Expertise, Trustworthiness, Tailoring, Commercial Implications, and Amateurism).  Below, I present comparisons for items in each scale.  I highlighted items that deviated from the 1999 values by more than 0.25 (without the original data, I couldn’t do much more in-depth comparison), but this might give some rough idea of which elements are more important now than they were in 1999 (red) and which elements are less important (purple):

Elements related to the "Real-World" feel scale are rated similarly from 1999 to 2009.

One difference is that people seem more critical of long download times than in the past.

For some reason, it seems that displaying an award helps your site's credibility more than 10 years ago, and that providing a lot of news stories matters less.

Stating your policy on content and ending in ".org" are more important to people now - could this be a cultural shift in response to sites like Wikipedia? Overall, it seems as if links out to other sites matter less for credibility now.

Email confirmations are more important now than in 1999.

Outside advertisements and an e-commerce focus matter more for credibility now than in the past. People are paying less attention to the commercial purpose of sites, as well as the number of ads and their integration with content (Google is a favorite site for many, after all.)

People seem to notice domain name mismatches now (more public knowledge of phishing/identity theft now?), but less attention to multi-lingual sites.

People seem more willing to trust free financial sites now than in the past.

Anyways, as we might have guessed, the answers from 2009 seem to match the answers from 1999 pretty well.  The elements that made things highly credible or highly ‘un-credible’ in the past seem to have remained constant, and those which didn’t matter then seem not to matter too much now.  Some interesting elements are noted in the captions with some rough conjectures as to why some of them might be trending the way they are.

The rest of the Fogg paper focused on characterizing differences in scale responses due to demographic differences between participants, but I found that part of the study less convincing as they averaged over the positively and negatively phrased elements on each scale (which I think makes the interpretation somewhat confusing).

Now, back to our two questions:

1) It looks as if using Mechanical Turk, we were able to get reasonable answers that fairly closely matched those from the original study.  Total price tag: 327 response * $0.05 = $16.35 paid to participants (plus Amazon’s cut), which made this study about $13,980 cheaper than the original one.

2) We see above a few small changes in what makes web sites credible, but overall, people are looking at the same things, meaning that we should continue taking the same factors into account when designing websites. I’ve made some guesses as to why these elements may have changed over time, but I’m curious to hear what you think, so leave a comment!

The data collected and the major points were fairly straightforward:

Participant Demographics:

• Age: Mean = 25.6, SD = 8.0
• Education: Almost all were mid-college or post-college (and about 1/6 post-graduate study).
• Usage: Most (56/78) reported specifically personal usage, and only 2 subjects reported specifically professional usage (14 indicated both, however).

Tools Used:

• Vast majority listed Facebook (61/78) – this was unsurprising (also, Facebook Stream listed as first example of an “activity stream” in survey instructions.)
• Wide Twitter usage (41/78) was surprising, however.  Past experiencing with polling for Twitter-related topics on MTurk had resulted in low yield.  Perhaps this is due to the crazy upswing in Twitter sign-ups over the past few months?
• Other than MySpace (16/78), tools such as LinkedIn, Yammer, FriendFeed, and others were barely listed, indicating either that these tools are not widely used or that people do not consider some of these to be activity streams.

Functions Served:

• Note: These responses were loosely categorized by me – this was not intended to be a rigorous academic study, but rather a glimpse into usage of these tools.
• Status(33/78), Communication(32/78), and Information(19/78) were listed as the most common functions served.
• Responses also demonstrated a wide variety of usage, however, including some less anticipated uses such as Journaling.  Perhaps this speaks somewhat to the flexibility of these tools and the ability that users have to adapt them to their own needs.

Feature Requests / Improvements:

• These are exact quotes from participants (again, loosely grouped into categories by me – no cross-coding was done).

As you can see, the “Summary” was really just a reminder about what people said regarding potential improvements, but I thought this was really the most interesting part.  It’s interesting that most of the things that people asked for were things that are either available or which could be easily made available by new activity stream client applications, so there may be a lot of low-hanging fruit out there for application developers.

I’d be curious to see if any of you has done (or seen) similar research regarding Twitter, Facebook, or other activity streams (whether on MTurk or otherwise) and if you found similar or different trends.  If you are interested in clarification, more details, or discussion about any of the points brought up here, the comments section awaits.

The ability to outsource data collection and have it populate a formatted spreadsheet in near real-time is fairly impressive (speed will probably differ on attractiveness of the task).  Often, the steps required to set up a MTurk task can take as long as the data collection itself, and afterwards you are left exporting to an unformatted Excel sheet.  Making sure to name data-labels correctly, populating data fields to randomize among workers, and other tasks can further complicate things; judging from the demo video, Smartsheet seems to handle these parts of the process fairly easily.

Unfortunately, good service doesn’t come free, and using the Smartsourcing feature requires signing up for a paid-account as well as the Smartsourcing fee on top of that, so I didn’t get the chance to try it out, but I would be extremely curious to hear about anyone’s experience with the service if they do try it.