Now, don’t get me wrong – I know that I have a long way left to go until people start calling me Dr. Kairam. At least for piano players, Ericsson theorized that 10,000 hours was the required amount of time to gain expertise, and I had always figured that PhD programs were around 5 years long for that very reason (40 hours/week * 50 weeks/year * 5 years = 10,000 hours, though it seems that some of us may become ‘double-experts’ by the time we’re done!). However, we’re also expected to complete some great research before we’ve finished the program; while I’ve done some research so far that I think is pretty good, I don’t think I’ve had any insights yet that I would consider ‘great’. As a result, it’s become difficult to shake the nagging doubt that perhaps I won’t get there.

Just as I was beginning to hit a low point, however, I came across this great video of radio host Ira Glass:

In case you don’t want to watch, he starts off by saying:

“Nobody tells this to people who are beginners, and I really wish someone had told me…All of us who do creative work, we get into it because we have good taste…But there’s a gap – that for the first couple years you’re making stuff, what you’re making isn’t so good…it’s trying to be good, it has ambition to be good, but it’s not quite that good. But your taste, the thing that got you into the game…is still killer. And your taste is good enough that you can tell what you’re making is kind of a disappointment to you…A lot of people never get past this phase…they quit.”

Inspired by this quote, I’ve decided to try and implement two policies to help foster my own creativity in research (as well as some other areas where I’m often creatively blocked, including songwriting and posting on this blog).

1. Repetition, Repetition, Repetition

Glass continues later in the video with the advice:

“The most important possible thing you can do is do a lot of work. Do a huge volume of work. Put yourself on a deadline so that every week or every month you know you’re going to finish one story…because it’s only by going through a volume of work that you’re actually going to catch up and close that gap and the work you’re making will be as good as your ambitions.”

Fostering creativity through repetition is evident in the insights gained from psychologist Keith Sawyer‘s interviews of winners of the New Yorker cartoon caption contest. According to his research, “the ‘sudden flash of insight’ is largely a myth”; instead, creative ideas ‘emerge over time’ through ‘hard work and constant revision’. Specifically, he says:

“Cartoon contest winners usually generate lots of captions. Studies have shown that quantity breeds quality – what I call the productivity theory, because high productivity corresponds to high creativity. When the famous physicist Freeman Dyson was asked how to generate good ideas, he said, ‘Have a lot of ideas, then throw out the bad ones.’ “

An important element in following this advice is reminding myself that I don’t have to publish everything I produce. If a project fails but spurs new ideas and helps me gain necessary skills, then I should view it as a success. If a song or blog post never quite comes together, it may inspire something better down the line. The important thing is to rehearse the process of crafting an idea, executing it, and committing it to paper so that I get practice with the creative part of the process. Regarding the process itself, this brings me to my second point:

2. Honor My Ideas

I draw my inspiration for this second policy from Lady Gaga, an artist who I view to be consistently creative. Near the end of GagaVision, episode 43, she describes her creative process:

Transcribed:

The creative process is approximately 15 minutes of vomiting my creative ideas…And then I spend days, weeks, months, years fine-tuning, but the idea is that you honor your vomit. You have to honor your vomit – you have to honor those 15 minutes.

While it sounds silly (and a little gross), I found these thoughts to be very instructive. I think that while I often have ideas that are creative or ‘out-there’, my internal filter shuts them down before I ever get a chance to examine whether or not they are viable. By committing your ideas to paper as soon as you have them, you can circumvent this filtering process so that those ideas don’t get lost. As Dyson said above, having a lot of ideas is a first step towards having good ideas.

As I’ve been taking the Caltrain to Stanford more often these days (in no way motivated by my spotting a sign for $4.99/gallon gas last week), I’ve decided to implement a policy of spending each morning train ride just throwing ideas on paper. Whether it’s lyrics to a song, thoughts for a blog post, or ideas for research, by forcing myself to just ‘vomit up’ whatever’s in my head, I am hoping that this deliberate practice at creativity will result in more ideas, and thus more good ideas, getting past my filter. In fact, that is actually how I put this blog post together, so let’s see if it keeps working.

If you try these or discover other methods for fostering your own creativity, share your experience in the comments!

Here are some brief notes on the presentation (to avoid the inevitable TL;DR comments):

• Data used was from a study I ran on Mechanical Turk looking at whether the tool WikiDashboard helps people to make different judgments about the credibility of Wikipedia articles.
• Participants placed in 1 of 3 conditions: (WO = Wiki Only, WH = Wiki + the History Page, WD = Wiki + WikiDashboard)
• Articles varied with respect to presumed quality and presumed controversy.
• Using non-parametric tests was fairly straightforward, but none were all that powerful (able to help find interaction effects – one main hope of the study would be to find an interaction between group and quality).

Anyways, this presentation is not supposed to be an expert statistics guide – rather, it represents the results of my research in trying to solve this problem (again, I’m very much not a statistics expert). There are surely many other ways to address the problem, and I would appreciate hearing from others who have tried attacking Likert items for their studies. I am continuing to analyze the data and may post some results in the near future.

As any parent or educator will likely tell you, the behavior of teens closely linked in a social network will often display many similarities: teens who smoke or drink, for instance, are often friends with other teens who smoke or drink. By establishing and tracking the spread of these behaviors scientifically, we can gain a greater understanding of the mechanisms at work and perhaps harness them to help spread healthy behaviors instead of unhealthy ones.

When we look at two teens who share a common behavior pattern (healthy or unhealthy), we must ask ourselves: Did they become friends because of their similar behavior (selection), did their behavior become similar as a result of being friends (influence), or was there some third factor at work which influenced them both separately (confounding factors)? One simple way to attempt to answer this question is through a longitudinal study, where data is collected for the same group of subjects at multiple times. By looking at the co-evolution of the social network and the behavior network, we can parse out the role that each of these factors plays. Here, I wanted to briefly discuss a few studies which have employed social network analysis and longitudinal data collection to gain a better understanding of how unhealthy behaviors can spread amongst teens.

The first is a study by Ennett and Bauman (PDF) which examines smoking as a function of position in the social network.(1) Specifically, they name three classes of social network patterns: cliques (a small group of at least three adolescents whose primary friendships are with each other), liaisons (adolescents who maintain multiple friendships without being in a particular friendship clique), and isolates (adolescents who have relatively few friendships with others.) Past research showing that cliques tend to share smoking behaviors leads many people to the assumption that smoking is a primarily peer group phenomenon.  However, after looking at data from 1,092 students collected across 5 schools over 1 year (from the start of 9th grade to the start of 10th grade), the authors found that smoking was far more common among isolates (17-40% across schools) than among clique members (4-16%).  Additionally, within the 87 cliques identified, they found that smokers tended to associate in the same cliques, with the majority of cliques composed entirely or almost entirely of non-smokers. In looking at the roles played by influence and selection, they indicate that both processes contributed equally to similarity in smoking behaviors for clique members, though they do not discuss how they performed their analysis.

The next paper described a 2009 study by Mercken, et al. (PDF) which utilizes a “stochastic actor-based model” to help in separating the roles of influence and selection “by simultaneously representing changes in friendship network structure and changes in smoking behavior among adolescents.” (2) In this study, they interviewed 1326 subjects from 11 Finnish schools 4 times over the course of 30 months (starting at the beginning of 7th grade). Each time, they asked about their friendship ties, their smoking behavior, that of their families, and their alcohol consumption. They found that adolescents who smoked more had a tendency to choose friends who likewise scored high on smoking behavior. Adolescents who smoke less than one cigarette per week were most likely to make friends with classmates who don’t smoke at all, while the most attractive potential friends for those who smoke one or more cigarettes per week were those who smoked at the highest rate. The authors did not report findings regarding the data collected on alcohol consumption, which hints at the fact that the patterns of spread may be different for different behaviors.

A natural question at this point is: are all friends created equal? Most of us growing up had a “best” friend in addition to our peer group. A 1997 study by Urberg, et al. (PDF) attempts to parse out the influence of close friends vs. that of one’s peer group as it pertains to both cigarette smoking and alcohol use.(3) In this study, they collected data from 1,028 Mid-western school-children in the 6th, 8th, and 10th grades; data was collected in two waves, once in the Fall and once in the Spring, and included assessments of friendship ties as well as cigarette and alcohol use. Interestingly, they found that it was smoking behavior of the peer group and not the close friend which predicted a transition into cigarette use, while it was the drinking behavior of the close friend and not the peer group which predicted a transition into alcohol use. They also found that those who have tried cigarettes or alcohol are more likely to know current users than those who have not (echoing the Mercken findings above).

Finally, because I couldn’t close a post on social networks and health without a Christakis/Fowler study, I wanted to mention a study from March of this year (2010) from Mednick, Christakis, and Fowler (PDF), which examined the interaction of two separate behaviors–low sleep and drug use–within a social network. (4) Looking at a sample of 8,349 adolescents from the ADD Health Data Corpus, they presented a number of interesting findings. First, they found that an individual’s behavior is correlated with the behavior of others in their network up to 4 degrees away. In the case of sleep, an individual was 29% more likely to sleep 7 hours or less if they had a friend who sleeps less than 7 hours; a friend of a friend correlated with a 17% increase, all the way down to a 5% increase for the friend of a friend of a friend of a friend. In the case of marijuana use, a direct connection to a user resulted in a 190% increase in the likelihood of use, while a 4th-degree connection still correlated with a 11% increase in use. Another interesting finding was that individuals central in the network were more likely to sleep less, with a two standard-deviation increase in centrality increasing the probability of sleeping 7 hours or less by 13% (controlling for other factors). Finally, they report on the interrelation between these behaviors, claiming that having a friend who slept 7 hours or less actually correlated with a 19% increase in smoking marijuana.

These 4 papers served as a useful introduction to both the methods of social network analysis and some of the interesting findings as they pertain to health behaviors and teens (across a number of behaviors – sleep, smoking, drinking, drugs). It is important for educators and health professionals to have an understanding of the social mechanisms as these will likely be a critical factor in preventing unhealthy behaviors from spreading amongst teens and persisting in their lives. Perhaps these same mechanisms can be used to spread positive behaviors such as exercise and civic-mindedness. In addition, it will be interesting to see how methods like these can be applied on a larger scale to Twitter or Facebook-sized social network corpora to track the spread of behaviors, ideas, diseases, and more across entire states or countries.

References:

1. Ennett, S.T. and Bauman, K.E. (2000). Adolescent social networks: Friendship cliques, social isolates, and drug use risk. In Hansen, W.B., et al. (eds) Improving prevention effectiveness. Tanglewood Research, Inc. Greensboro, NC.
2. Mercken, L., et al. (2009). Dynamics of adolescent friendship networks and smoking behavior. Social Networks.
3. Urberg, K.A., Değirmencioğlu, S.M., and Pilgrim, C. (1997) Close Friend and Group Influence on Adolescent Cigarette Smoking and Alcohol Use. Developmental Psychology, vol 33(5), pp. 834-844.
4. Mednick, S.C., Christakis, N.A., Fowler J.H. (2010) The Spread of Sleep Loss Influences Drug Use in Adolescent Social Networks. PLoS ONE vol 5(3): e9775.

Also, I want to thank Sarita Yardi and Vladimir Barash for directing me towards some of these papers.

• Identifying the features that underlie “tag quality” (e.g. Sen, et al. (2007), Zhang, et al. (2009))
• Topic-based approaches for information retrieval from tagged collections (e.g. Zhou, et al. (2008))
• Graph-based algorithms for ranking based on user tags (e.g. Hotho, et al. (2006), Noll, et al. (2009))

I decided to try a little Mechanical Turk study to see if I could spot some differences between tags generated by experts and those generated by novices. I had each Turker read 1 of 5 web pages (on the topic of “enterprise 2.0 mashups”) and enter 5 tags which they thought would be useful for bookmarking the page (either for themselves or others). I also asked them to rate how familiar they were with the subject matter (“Not at All”, “Slightly Familiar”, “Somewhat Familiar”, and “I am an Expert”).

As a game, I thought it would be interesting to post some of the responses to see how easy it was to identify which tags were generated by people who rated themselves as “experts” vs. “non-experts”. I took all of the tags generated by each expertise group, cleaned them up for minor spelling mistakes and typos (e.g., “applciation” > “application”) and generated a tag cloud using Wordle, where the tag size corresponds to the frequency of use of that word (all other factors, such as positioning and color, are purely stylistic).

For the following URL – http://www.soamag.com/I18/0508-1.php – can you identify which tag cloud belongs to which of these groups: “Not at All (Familiar)”, “Slightly Familiar”, and “Somewhat Familiar” (there was a 4th category of “I am an Expert”, but nobody rating this URL classified themselves this way):

Tag Cloud 1 (N = 17)

Tag Cloud 2 (N = 16)

Tag Cloud 3 (N = 14)

If you have any idea which tag cloud is which, please feel free to post your guess in the comments! I’d be extremely curious to see why people guessed the way that they did. I am actually currently in the process of having some Turkers do the same thing; if you are curious about the answers, come back for my follow-up post where I post the correct answers, as well as the results of the Mechanical Turk evaluation of the tag cloud.

The paper, by Aardvark’s Damon Horowitz and Stanford’s Sep Kamvar, focuses mostly on the architecture of the Aardvark system, from the external representations with which users interact to the internal ranking algorithms on which the system runs. Below, I present a short summary of what they report, focusing on the elements I found most interesting:

The Basic Model: Aardvark’s scoring function is similar to PageRank in that both utilize two primary, but somewhat independently considered components: relevance and quality.

• Relevance in the Aardvark model pertains to the probability that a particular user i can answer the given question q based on the identified topics contained in t.
• Quality in the Aardvark model pertains to the overall probability that a user i can return a satisfactory answer to another user j, regardless of the question.

Indexing Topics: Aardvark computes the relevance score by calculating a distribution of knowledge over topics known by the user using the following sources (keyword-y sounding italicized terms are for convenience only and are not used in the paper):

• Explicit Prompting at sign-up for three “starter” topics about which the user has expertise.
• Social Prompting of a user’s friends to provide topics about which they trust the user’s opinion.
• Structured Parsing of the online profile pages connected to Aardvark by the user (e.g. “Interests” on a Facebook profile).
• Unstructured Parsing of the users’ online homepage, blog, or status updates using a linear SVM to extract overall subject area and a named entity extractor to extract more specific topics.

Indexing Connections: Aardvark computes the quality score by building a set of weighted connections between users using characteristics ranging from social proximity to similarities in demographics or behavior, such as:

• Social Connections either in the form of explicitly defined “friend” connections or implicit “network” connections, such as both being part of the Stanford network.
• Demographic Similarity, which likely includes age, gender, and location based on profile information collected by Aardvark.
• Profile Similarity, which seems to include similar movies and other items which might be listed on other profiles, such as Facebook.
• Vocabulary Match, which they explain with the example of “IM Shortcuts” (i.e. I assume this means it is based on the language you use to interact with Aardvark, but I am unsure.)
• Chattiness and Verbosity Match, which relate to frequency and length of messages used when interacting with Aardvark.
• Politeness Match, which basically seems to mean whether or not say “Thanks!” or not.
• Speed Match, which is a measure of responsiveness to other users.

Analyzing Questions: While all of the other components are pre-computed, this part is computed at question time (obviously). The utilize a number of classifiers to classify the question and then a set of mappers to map the question to a set of topics, noting that “the role of the Question Analyzer…is simply to learn enough about the qeustion that it may be sent to appropriately interested and knowledgeable human answerers”. Here are the classifiers they list (with the names used in the paper):

• NonQuestionClassifier: Determines if input is a valid question.
• InappropriateQuestionClassifier: Determines if input is obscene, spam, or otherwise unsuitable for asking.
• TrivialQuestionClassifier: Determines if input is a simple factual question (examples given: “What time is it now?”, “What is the weather?”). If so, the user gets an automatically generated answer via traditional web search.
• LocationSensitiveClassifier: Determines if the question contains location information; if it does, it passes that information along to the Routing Engine

• KeywordMatchTopicMapper: Checks for string matches against user profile topics (the mapper attempts to classify meaningful vs. spurious matches).
• TaxonomyTopicMapper: Classifies question text using an SVM trained on an “annotated corpus of several million questions” (where did they find that?)
• SalientTermTopicMapper: Extracts salient phrases using a noun-phrase chunker and tf-idf and finds “semantically similar user topics”.
• UserTagTopicMapper:Utilizes tags explicitly provided by the asker or other answerers and maps them to user topics.

This description of the routing algorithm comprises the main function of the paper. After some more description of how users interact with the system, the authors provide some interesting data collected over the past several months of use (from the beta launch in March 2009 until October 2009).  Here’s a quick run-down of the more interesting facts that they presented:

• Strong User Growth: As of October 2009, they reported 90,361 user accounts, and users appear to be remaining active (in the study period, over 1/2 the users actively generated content and over 2/3 of the users passively participated).

User Growth on Aardvark (graph taken from the paper).

• Higher Query Contextualization: Aardvark queries average 18.6 words in length while the average query length reported for web search is between 2.2 and 2.9 words (citing previous comparison and characterization studies).  They further state that “98.1% of questions are unique”, though I am unsure as to how exact they are being about matching (I am sure the question “What’s a great restaurant in SF” has been asked 1000 times in different forms). In addition, they report from manual scoring of 1000 randomly selected questions that 64.7% of questions asked have a subjective element, with advice about travel, restaurants, and products being specifically popular.
• Fast, High-Quality Answers: They report that 87.7% of questions get answers and 57.2% received an answer within 10 minutes. They report that 70.4% of answers receiving feedback are rated as “good” and only 15.5% are rated as “bad”. Interestingly, they observe a notable difference in feedback on answers from users within the asker’s social network (76% rated as food) and outside the asker’s network (68% rated as good).

Questions on Aardvark (chart taken from the paper).

Overall, I really enjoyed reading this paper. After using Aardvark for over a year now, it was really interesting to get to peer inside and see how the system works, and a lot of great details were provided about the ranking engine.

One place where I feel that the authors missed the mark was in the cursory side-by-side evaluation which pitted Aardvark against Google for a set of 200 questions randomly selected from the Aardvark system. They report that 71.5% of the questions studied were answered successfully on Aardvark, while 70.5% of the questions were answered successfully on Google. This comparison seems mostly useless as the questions, having been pulled from the Aardvark system in the first place, are ones that were specifically chosen because they are better adapted to what is being called ‘social search’. This comparison left me desirous of more investigation into two main questions.

“What makes a search engine ‘social’ in the first place?”

The distinction between social and non-social is extremely murky, something Brynn and I discovered when working on our Social Search paper. It has been argued before (one small example here) that Google’s PageRank algorithm is inherently social, as it aggregates information provided by people (links to one another) to rank results. However, it is clear that something seems categorically different between Google and what people perceive to be ‘social search’. When it comes down to it, even though everyone is excited about Google’s forays into “Social Search”, there’s nothing all that fundamentally different about Google indexing your blog and your tweets than any other documents extant on the web.

To me, it seems that the key difference is really the change in the direction of interaction. While Google takes a query (question) and compares it against traces of discussion about that question from the past (web documents), systems perceived as ‘social’ take a question and attempt to generate new answers in the future. This change in direction is what allows for the higher context that makes ‘social’ search answers so much more rich (at least for some questions.)  Perhaps we need a different word to define this phenomenon – ‘real-time search’ seems to get at it more, but has its own problems.  Perhaps something like ‘generative search’? I really don’t know.

“Why do we need a social search engine at all?”

This one seems like the best fodder for a follow-up study by Aardvark. While they do provide a rough breakdown of the types of questions asked on Aardvark (see pie chart above), I think that a comparison might have been much more interesting if they had looked at a variety of classes of user needs and had compared the relative efficacy of searching on Aardvark and a traditional search engine such as Google. It is clear that ‘social’ will work much better for some needs and much worse for others, but up to this point, people who talk about social search always seem to use the same types of examples (travel, restaurants, and products, for instance). It would be great to get a clear idea over a wide range of needs and use cases where systems such as Aardvark can provide benefits over existing tools.

Anyways, for those of you interested in ‘social search’ and search systems, I encourage you to read this paper and tell me your thoughts!

Most research effort does not produce what is thought of as a traditionally publishable result.  That doesn’t mean, however, that nothing was gained by conducting the research.  These results, whether they are failures or merely perplexing, can provide valuable insights into open problems and prevent other researchers from duplicating work.

We have started a journal that focuses on serendipitous (I have no idea why this worked) and unexpected (it seems like this technique should work on this problem but it doesn’t) results.  The goal is to provide a venue for the dissemination and discussion of ideas and to enable more efficient research.

The Journal of Serendipitous and Unexpected Results (JSUR) is an open-access forum for researchers seeking to further scientific discovery by sharing surprising or unexpected results. These results should provide guidance toward the verification (or negation) of extant hypotheses.  JSUR has two branches, one focusing on computational sciences and the other on the life sciences.  JSUR submissions include, but are not limited to, short communications of recent research results, full-length papers, review articles, and opinion pieces.

Recently, we launched the beta version of the journal site at http://jsur.org .  We would love to get your feedback and even better, a submission for the first issue.

To get the journal started, we’re looking to collect a large number of short (2-4 page) reports. I know you have something to publish.

Please help us spread the word and forward this information to interested colleagues.

Looking at the author guidelines, it looks as if submissions will be reviewed both by an editorial board and through a peer review process.  In addition, an interesting note is that all articles will be open-access and Creative-Commons licensed.  I’ll definitely be happy to read some of the articles that come through here once they start publishing.

Upon reading this study, I noticed two things:

1) The original experiment solicited participants by offering to donate $10 to charity for their time; resulting in a net cost of at least $14K, raising the second question: Given the growth of sites like Mechanical Turk, can we get comparable results for less money?

2) The data was originally collected in December 2009 1999, which is almost exactly 10 years ago. Content on the web and our interactions with it have changed a great deal since then – I mean, in 1999, Google had only existed for a year, blogs were still relatively uncommon (LiveJournal had just started), and Wikipedia didn’t even exist yet! This raised the question: Given that the average Internet user now has a far greater amount of experience navigating the Web, should we expect the responses to be different 10 years later?

I decided to explore both of these questions through a survey on Mechanical Turk; in December 2009, I posted a HIT to Mechanical Turk replicating the original study as closely as possible. The 51 statements about web site elements were available from Fogg’s original paper, the same 7-point Likert scale was used, and the survey items were randomized.  I paid $0.05 per HIT, and I ended up getting 327 responses, with none thrown out due to quality.

An initial, high-level examination of the responses showed that they actually matched the 1999 data fairly well.  Average Likert ratings for each item correlated highly with average ratings for the same items in the 1999 data with R^2 = 0.96.  One difference in the data was that answers were compressed (closer to 0) overall, so for the purposes of comparison, I transformed the 2009 data using the transformation (1.1677X – 0.0003) to match the 1999 data (note that this has no effect on the correlation coefficient).

The first analysis in Fogg, et al. was to separate the various elements into 7 “scales” using factor analysis (Real-World Feel, Ease of Use, Expertise, Trustworthiness, Tailoring, Commercial Implications, and Amateurism).  Below, I present comparisons for items in each scale.  I highlighted items that deviated from the 1999 values by more than 0.25 (without the original data, I couldn’t do much more in-depth comparison), but this might give some rough idea of which elements are more important now than they were in 1999 (red) and which elements are less important (purple):

Elements related to the "Real-World" feel scale are rated similarly from 1999 to 2009.

One difference is that people seem more critical of long download times than in the past.

For some reason, it seems that displaying an award helps your site's credibility more than 10 years ago, and that providing a lot of news stories matters less.

Stating your policy on content and ending in ".org" are more important to people now - could this be a cultural shift in response to sites like Wikipedia? Overall, it seems as if links out to other sites matter less for credibility now.

Email confirmations are more important now than in 1999.

Outside advertisements and an e-commerce focus matter more for credibility now than in the past. People are paying less attention to the commercial purpose of sites, as well as the number of ads and their integration with content (Google is a favorite site for many, after all.)

People seem to notice domain name mismatches now (more public knowledge of phishing/identity theft now?), but less attention to multi-lingual sites.

People seem more willing to trust free financial sites now than in the past.

Anyways, as we might have guessed, the answers from 2009 seem to match the answers from 1999 pretty well.  The elements that made things highly credible or highly ‘un-credible’ in the past seem to have remained constant, and those which didn’t matter then seem not to matter too much now.  Some interesting elements are noted in the captions with some rough conjectures as to why some of them might be trending the way they are.

The rest of the Fogg paper focused on characterizing differences in scale responses due to demographic differences between participants, but I found that part of the study less convincing as they averaged over the positively and negatively phrased elements on each scale (which I think makes the interpretation somewhat confusing).

Now, back to our two questions:

1) It looks as if using Mechanical Turk, we were able to get reasonable answers that fairly closely matched those from the original study.  Total price tag: 327 response * $0.05 = $16.35 paid to participants (plus Amazon’s cut), which made this study about $13,980 cheaper than the original one.

2) We see above a few small changes in what makes web sites credible, but overall, people are looking at the same things, meaning that we should continue taking the same factors into account when designing websites. I’ve made some guesses as to why these elements may have changed over time, but I’m curious to hear what you think, so leave a comment!

The data collected and the major points were fairly straightforward:

Participant Demographics:

• Age: Mean = 25.6, SD = 8.0
• Education: Almost all were mid-college or post-college (and about 1/6 post-graduate study).
• Usage: Most (56/78) reported specifically personal usage, and only 2 subjects reported specifically professional usage (14 indicated both, however).

Tools Used:

• Vast majority listed Facebook (61/78) – this was unsurprising (also, Facebook Stream listed as first example of an “activity stream” in survey instructions.)
• Wide Twitter usage (41/78) was surprising, however.  Past experiencing with polling for Twitter-related topics on MTurk had resulted in low yield.  Perhaps this is due to the crazy upswing in Twitter sign-ups over the past few months?
• Other than MySpace (16/78), tools such as LinkedIn, Yammer, FriendFeed, and others were barely listed, indicating either that these tools are not widely used or that people do not consider some of these to be activity streams.

Functions Served:

• Note: These responses were loosely categorized by me – this was not intended to be a rigorous academic study, but rather a glimpse into usage of these tools.
• Status(33/78), Communication(32/78), and Information(19/78) were listed as the most common functions served.
• Responses also demonstrated a wide variety of usage, however, including some less anticipated uses such as Journaling.  Perhaps this speaks somewhat to the flexibility of these tools and the ability that users have to adapt them to their own needs.

Feature Requests / Improvements:

• These are exact quotes from participants (again, loosely grouped into categories by me – no cross-coding was done).

As you can see, the “Summary” was really just a reminder about what people said regarding potential improvements, but I thought this was really the most interesting part.  It’s interesting that most of the things that people asked for were things that are either available or which could be easily made available by new activity stream client applications, so there may be a lot of low-hanging fruit out there for application developers.

I’d be curious to see if any of you has done (or seen) similar research regarding Twitter, Facebook, or other activity streams (whether on MTurk or otherwise) and if you found similar or different trends.  If you are interested in clarification, more details, or discussion about any of the points brought up here, the comments section awaits.