Now, don’t get me wrong – I know that I have a long way left to go until people start calling me Dr. Kairam. At least for piano players, Ericsson theorized that 10,000 hours was the required amount of time to gain expertise, and I had always figured that PhD programs were around 5 years long for that very reason (40 hours/week * 50 weeks/year * 5 years = 10,000 hours, though it seems that some of us may become ‘double-experts’ by the time we’re done!). However, we’re also expected to complete some great research before we’ve finished the program; while I’ve done some research so far that I think is pretty good, I don’t think I’ve had any insights yet that I would consider ‘great’. As a result, it’s become difficult to shake the nagging doubt that perhaps I won’t get there.

Just as I was beginning to hit a low point, however, I came across this great video of radio host Ira Glass:

In case you don’t want to watch, he starts off by saying:

“Nobody tells this to people who are beginners, and I really wish someone had told me…All of us who do creative work, we get into it because we have good taste…But there’s a gap – that for the first couple years you’re making stuff, what you’re making isn’t so good…it’s trying to be good, it has ambition to be good, but it’s not quite that good. But your taste, the thing that got you into the game…is still killer. And your taste is good enough that you can tell what you’re making is kind of a disappointment to you…A lot of people never get past this phase…they quit.”

Inspired by this quote, I’ve decided to try and implement two policies to help foster my own creativity in research (as well as some other areas where I’m often creatively blocked, including songwriting and posting on this blog).

1. Repetition, Repetition, Repetition

Glass continues later in the video with the advice:

“The most important possible thing you can do is do a lot of work. Do a huge volume of work. Put yourself on a deadline so that every week or every month you know you’re going to finish one story…because it’s only by going through a volume of work that you’re actually going to catch up and close that gap and the work you’re making will be as good as your ambitions.”

Fostering creativity through repetition is evident in the insights gained from psychologist Keith Sawyer‘s interviews of winners of the New Yorker cartoon caption contest. According to his research, “the ‘sudden flash of insight’ is largely a myth”; instead, creative ideas ‘emerge over time’ through ‘hard work and constant revision’. Specifically, he says:

“Cartoon contest winners usually generate lots of captions. Studies have shown that quantity breeds quality – what I call the productivity theory, because high productivity corresponds to high creativity. When the famous physicist Freeman Dyson was asked how to generate good ideas, he said, ‘Have a lot of ideas, then throw out the bad ones.’ “

An important element in following this advice is reminding myself that I don’t have to publish everything I produce. If a project fails but spurs new ideas and helps me gain necessary skills, then I should view it as a success. If a song or blog post never quite comes together, it may inspire something better down the line. The important thing is to rehearse the process of crafting an idea, executing it, and committing it to paper so that I get practice with the creative part of the process. Regarding the process itself, this brings me to my second point:

2. Honor My Ideas

I draw my inspiration for this second policy from Lady Gaga, an artist who I view to be consistently creative. Near the end of GagaVision, episode 43, she describes her creative process:

Transcribed:

The creative process is approximately 15 minutes of vomiting my creative ideas…And then I spend days, weeks, months, years fine-tuning, but the idea is that you honor your vomit. You have to honor your vomit – you have to honor those 15 minutes.

While it sounds silly (and a little gross), I found these thoughts to be very instructive. I think that while I often have ideas that are creative or ‘out-there’, my internal filter shuts them down before I ever get a chance to examine whether or not they are viable. By committing your ideas to paper as soon as you have them, you can circumvent this filtering process so that those ideas don’t get lost. As Dyson said above, having a lot of ideas is a first step towards having good ideas.

As I’ve been taking the Caltrain to Stanford more often these days (in no way motivated by my spotting a sign for $4.99/gallon gas last week), I’ve decided to implement a policy of spending each morning train ride just throwing ideas on paper. Whether it’s lyrics to a song, thoughts for a blog post, or ideas for research, by forcing myself to just ‘vomit up’ whatever’s in my head, I am hoping that this deliberate practice at creativity will result in more ideas, and thus more good ideas, getting past my filter. In fact, that is actually how I put this blog post together, so let’s see if it keeps working.

If you try these or discover other methods for fostering your own creativity, share your experience in the comments!

The presentation ends with a quick intro in the JavaScript code (from the Iterative Text Improvement example on the TurKit website) and some hopefully helpful information to know when using the Java application that you can download to try out TurKit. If I survive the end of the quarter, I hope to get a post up with some TurKit tutorial tips and lessons learned. If you have questions about TurKit, let me know, and I’ll try to get them answered in the next post!

Based on this post and other comments, it is fairly clear that MG Siegler is on “Team Quora”, calling Quora “a great source of information like Twitter and Facebook and blogs themselves.” Having been a Quora user for several months, I have found the quality of answers on the site to be extremely high. These have ranged from the much-celebrated cases of high-profile individuals answering questions about topics pertaining to them (e.g. Netflix CEO Reed Hastings answering the question “How much does Netflix spend on postage every year?“) to the opportunities for creative individuals to answer questions in awesome and innovative ways (see Wavii programmer Erik Frey‘s answer to the question “Which animal has been used most frequently for a band name?“). I have also personally asked a number of questions and gotten timely and high-quality answers.

However, the key question here is whether the site will continue to be as useful as more and more people join. While many might argue that including more subject matter experts can only improve the site, one must also remember that this increased signal is only useful when it can be separated from the increased noise. Right now, Quora is a bit like <nerd alert>Flynn’s cave dwelling in the Outlands, where the information contained is only safe as long as the masses can’t get to it</nerd alert>; users can trust the answers they find because they often come paired with often famous or at least recognizable names and faces. Yahoo! Answers is a great example of how a Q&A site can decrease in quality with respect to both the questions and answers as it opens up (for a quick, possibly NSFWish laugh, check out “11 Stupid Questions from Yahoo Answers That Have Changed My Life“) Even if you develop the most robust social answer-quality-checker imaginable, the presence of thousands of stupid questions tagged with topic tags that direct them to your inbox is going to turn a lot of the quality answer providers away from the site. In some ways, I felt that Aardvark fell into that trap as it grew more popular, and I now find myself answering a lot of questions that involved identifying rashes.

I had this question about the possible perils of mainstream adoption in mind, when reading a NY Times blog post this week, in which David Pogue describes his first interactions with the site as “a descent into bafflement.” Among the parts of the site that he deems confusing are the login process, the task of adding connections and following topics, and the actual task of asking a question. For chronicling his confusion, however, he earned the following response from Siegler (on his personal blog):

Is this meant to be written from the perspective of a 95-year-old senile man?

Apparently, every site should be designed in a way so that it’s just like every other site that failed before it on the Internet. Makes perfect sense.

Prediction: he’ll love Quora in 12 months.

Now, I would consider David Pogue to be a relatively tech-savvy individual. He’s been blogging about Web/technology topics for over 10 years now, and I generally find his posts to be pretty interesting and insightful. If he is having this much difficulty using the site, I think that his frustrations really do say something about Quora’s current potential to reach beyond the geek crowd into the general public . And on some level, I think that the desire to mock him for this reflects an underlying recognition that the Silicon Valley influentials who currently use the site wouldn’t actually benefit from “regular people” using it, since this could potentially spell the end of Quora’s usefulness. Perhaps a better use of time might be to consider how to make the site more user-friendly and how to maintain the quality as new users arrive.

Studying sites like Wikipedia is a great way to examine how to maintain quality while scaling out to a broader audience. While it’s difficult to ascertain what actually makes Wikipedia work (in practice, if not in theory), many of the qualities inherent in Wikipedia are those identified in a 2005 paper [1] by J.M. Leimeister and colleagues at the Technische Universität München as factors which promote community success in an environment where trusted information is key. Some of these include exposing the identity of content providers, clearly establishing goals for the community, making member profiles available to other members, and providing various levels of anonymity, all of which are things which are built into Wikipedia’s core. Another feature recognized across the literature is recognition and rewards for contributors, something embodied in Wikipedia in the form of “barnstars“. Quora does a great job of exposing identity (which will likely make it much better than Yahoo! Answers), but I don’t believe that it adequately addresses these other elements. I think that the addition of “moderators” or other custodian roles for hyper-motivated users could be the kind of thing that keeps the Quora community in check, and I would be eager to see them roll something like that out before opening the site up to the general public (I think it’s technically still in beta, no?)

Quora is a great source of information for me in some of the same ways that Facebook and Twitter are. The key difference is that in those media, I can control what I see and who it comes from. If you want to imagine the utility of those sites without such controls, just imagine trying to sift through the real-time Twitter stream – it’s mind-numbing, to be kind (unless of course you are a Justin Bieber fan, in which case you should be delighted). The goals of those who currently enjoy Quora’s usefulness should be to help maintain that usefulness as the site grows. I’m not saying that this will be an easy task, but if it isn’t accomplished, I don’t think that anyone is going to love the site in 12 months.

[1] Leimeister, J.M., Ebner, W., & Krcmar, H. 2005: Design, implementation, and evaluation of trust-supporting components in virtual communities for patients. Journal of Management Information Systems 21, 4, 101-135.

For a mid-quarter assignment, I worked with Diana MacLean on a project related to visualizing social network patterns. The goal of the project was to build an interactive visualization which would help support exploration of the data and help surface interesting patterns. We decided to build the project using Protovis, a JavaScript-based graphical toolkit for data visualization built by Jeff and one of his students, Michael Bostock. We chose Protovis (rather than another toolkit like Prefuse or Flare) primarily because it seemed like the easiest way to post the visualization on the web when we were done.

In this project, we chose to examine methods for visualizing cross-posting behaviors of users of MedHelp, an Online Health Community (OHC). Users suffering from various conditions come to these communities to seek advice, brainstorm about symptoms and treatments, or offer each other emotional support. An interesting pattern identified by Diana was the prevalence of users who post primarily in one community (such as “Asthma”) posting in a different, secondary community (such as “Fertility”). We were particularly interested in such behavior, which we labeled “cross-posting”. According to recent research, identifying patterns in cross-posting behavior could potentially help discover previously unknown, or untested, medical links between symptoms. In the example given above, a large degree of cross-posting between “Asthma” and “Fertility” could suggest a potential link between the two conditions. In our visualization, we represent the cross-posting data as a graph, where nodes correspond to communities on the website and directed edges (A -> B) are weighted by the number of users who post primarily in community A who also post in community B. We utilized an adjacency matrix rather than a network layout because we felt that the edge-centric view would make it easier to spot trends and unusually strong or weak connections between communities.

The focus of the class was primarily on the design process, so if you are interested in how we came to our final version, you can check out our project page on the class website. Below, you can see a screenshot of what we produced. To try it out for yourself, click here, and the page will open in a new window.

Screenshot of our Interactive Visualization of Cross-Posting Behavior in the MedHelp Online Health Community Forums. (Note, the information section in the bottom right was added as a *.png and I'm having difficulty locating it right now, which is why it's missing in the version linked to.)

The top, left panel shows an adjacency matrix in which hue encodes the value of the cell (that is, the percentage of members whose primary community is ROW community, who posted in COLUMN community). Several mouseover and click techniques are enabled on the matrix. In the top right panel is a tag cloud of the most frequent words in the cross posts corresponding to the active cell. The words are stemmed for more accurate counting, although this occasionally makes them tricky to decipher. Tag clouds are generated (force layout diagram on a set of unconnected nodes, sized according to normalized word frequency) upon first mouseover; subsequent mouse-overs render the previously generated layout to avoid too much distracting motion. The bottom left panel contains the color scale for the matrix hue values. A highlight bar on the scale indicates the value of the active cell. Superimposed on this scale is a histogram of value frequency, which allows the user to note the commonness/scarcity of the active value in question.

Clicking a cell changes the interaction state so that the row and column of the current selected cell to be highlighted in red. (Note – you have to click and move to another cell first before the highlighting triggers.) This enables a quick comparison of the selected cell value with all other values for a.) cross posting originating from ROW community (to any COLUMN community), and b.) cross posts originating from any ROW community to the selected COLUMN community. This way, users can compare the significance of the selected cell value with all similar (in the sense that they share either a source or an origin community) values.

The project was an interesting foray into alternative techniques for visualizing interaction in a network. The idea of a network is so tied up in our minds with node-link diagrams, that it’s often hard to picture them any other way. Even if Data Visualization doesn’t become a large part of my later research, it’s extremely useful in helping generate different types of mental models for the same data. What other methods do you think might be more effective for the MedHelp data?

Note: Much of the description text is adapted from our project submission write-up, a large portion of which was done by Diana.

Here are some brief notes on the presentation (to avoid the inevitable TL;DR comments):

• Data used was from a study I ran on Mechanical Turk looking at whether the tool WikiDashboard helps people to make different judgments about the credibility of Wikipedia articles.
• Participants placed in 1 of 3 conditions: (WO = Wiki Only, WH = Wiki + the History Page, WD = Wiki + WikiDashboard)
• Articles varied with respect to presumed quality and presumed controversy.
• Using non-parametric tests was fairly straightforward, but none were all that powerful (able to help find interaction effects – one main hope of the study would be to find an interaction between group and quality).

Anyways, this presentation is not supposed to be an expert statistics guide – rather, it represents the results of my research in trying to solve this problem (again, I’m very much not a statistics expert). There are surely many other ways to address the problem, and I would appreciate hearing from others who have tried attacking Likert items for their studies. I am continuing to analyze the data and may post some results in the near future.

Poisson distribution. Poisson process. Poisson equation. Poisson kernel. Poisson regression. Poisson summation formula. Poisson’s spot. Poisson’s ratio. Poisson bracket. Euler-Poisson-Darboux equation. This is only a partial list, and yet it shows the degree to which Siméon-Denis Poisson’s work has impacted just about all branches of science. But what is so impressive is not the volume of his contributions but rather their depth, raising a puzzling question: How did Poisson manage to work simultaneously on so many quite different problems and yet stay sufficiently focused to offer deep and lasting contributions?

Well, we had a secret: a notebook and a tiny habit.

Each time Poisson encountered a problem he though fascinating, he would resist the temptation to savor it. He pulled out his notebook instead and made a note of it and promptly returned to the problem that had absorbed him before the interruption. Once he solved the problem at hand, he mulled over the list of problems scribbled in his notebook, then picking as his next challenge the one he found the most interesting.

Poisson’s little secret was lifelong, careful prioritizing.

So, this essentially describes the polar opposite of my work habits, which currently consist of frenetically switching from task to task to ensure that I complete none of them. I’m thinking of giving the priority list a try – has anybody tried a scheme like this and had success with it? Would be curious to hear your story!

Eugene Eric Kim: Strategy Program Manager

• If you include all of the component sites, Wikimedia is the 5th most accessed web-property in the world.
• 350M regular visitors, $10M in revenue, and only 35 employees.
• 45K active contributors (a term they use to indicate people who make 5 or more edits per month) on English Wikipedia.
• The country with the most visitors is actually Canada (which nobody in the audience guessed).
• Defined the Wikimedia Foundation mission with a Jimmy Wales quote: “Imagine a world in which every single person on the planet is given free access to the sum of all human knowledge.“

Trevor Parscal: Lead Front-End, UX Programs

• Trevor is the guy in charge of “basically everything you see” (wow!)
• Wikimedia research shows that people don’t find the software easy to use (duh), so they have launched the Usability Initiative.
• In fact, when they were testing with users, they had one user who took 20 minutes to figure out how to edit a page (and this wasn’t entirely out of the ordinary).
• Asking people what they wanted in the site proved not-so-successful, but having them try out a new Beta version and observing behavior was really fruitful.
• As of now, 84% of the people who opted into the Wikipedia Beta have stayed (almost 300K) people – (there was no mention of how to find the beta, btw).

Tomasz Finc: Engineering Program Manager

• Fundraising is done annually, between November and January.
• Amount raised: 2006 – $1m, 2007 – $2M, 2008 – $6M, and 2009 – $8.1M
• Most of their fundraising comes from small donations (contrary to usual trend of large donations for these types of efforts)
• Did a lot of A/B style testing to figure out how to optimize contribution – a lot of this is actually shared on the Wikimedia Blog.
• Adding Jimmy Wales’ plea increased the donations a LOT (so much that at first they thought the site was being attacked).
• The iPhone application and mobile gateway are both being developed by the community.
• The OLPC now has a full copy of the English Wikipedia on it.

As you can see, the talks basically focused on three elements: 1) Wikipedia is big and wants to get bigger, 2) Wikipedia is hard to use and wants to get easier, 3) Wikipedia relies a lot on the community. While there wasn’t much that was earth-shattering, each of these elements was pretty interesting – the idea that such a HUGE platform and vast amount of content can be supported by just 35 full-time employees and the contributions of the community is incredible, and speaks to the power that effective community management can bring. As Wikipedia is one of the greatest examples of social software and content production, it was great to get the opportunity to peer under the hood a little bit.

For some more information that may not have made it into this brain-dump, check out my live-tweet of the event here.

• Identifying the features that underlie “tag quality” (e.g. Sen, et al. (2007), Zhang, et al. (2009))
• Topic-based approaches for information retrieval from tagged collections (e.g. Zhou, et al. (2008))
• Graph-based algorithms for ranking based on user tags (e.g. Hotho, et al. (2006), Noll, et al. (2009))

I decided to try a little Mechanical Turk study to see if I could spot some differences between tags generated by experts and those generated by novices. I had each Turker read 1 of 5 web pages (on the topic of “enterprise 2.0 mashups”) and enter 5 tags which they thought would be useful for bookmarking the page (either for themselves or others). I also asked them to rate how familiar they were with the subject matter (“Not at All”, “Slightly Familiar”, “Somewhat Familiar”, and “I am an Expert”).

As a game, I thought it would be interesting to post some of the responses to see how easy it was to identify which tags were generated by people who rated themselves as “experts” vs. “non-experts”. I took all of the tags generated by each expertise group, cleaned them up for minor spelling mistakes and typos (e.g., “applciation” > “application”) and generated a tag cloud using Wordle, where the tag size corresponds to the frequency of use of that word (all other factors, such as positioning and color, are purely stylistic).

For the following URL – http://www.soamag.com/I18/0508-1.php – can you identify which tag cloud belongs to which of these groups: “Not at All (Familiar)”, “Slightly Familiar”, and “Somewhat Familiar” (there was a 4th category of “I am an Expert”, but nobody rating this URL classified themselves this way):

Tag Cloud 1 (N = 17)

Tag Cloud 2 (N = 16)

Tag Cloud 3 (N = 14)

If you have any idea which tag cloud is which, please feel free to post your guess in the comments! I’d be extremely curious to see why people guessed the way that they did. I am actually currently in the process of having some Turkers do the same thing; if you are curious about the answers, come back for my follow-up post where I post the correct answers, as well as the results of the Mechanical Turk evaluation of the tag cloud.

THE OBJECT OF THE AUTHOR in the following pages has been to collect the most remarkable instances of those moral epidemics which have been excited, sometimes by one cause and sometimes by another, and to show how easily the masses have been led astray, and how imitative and gregarious men are, even in their infatuations and crimes.

Charles Mackay – Memoirs of Extraordinary Popular Delusions and the Madness of the Crowds

Image courtesy of Marcos Vasconcelos (click for original)

I’ve been interested a bit in looking at how to help people find high-quality information on the web – recently, I have been exploring how to help people make better credibility judgments about the information they find. One paper I was reading, “Statement Map: Assisting Information Credibility Analysis by Visualizing Arguments” by Koji Murakami and others at the Nara Institute of Science and Technology in Japan, uses as a motivating example the recent movement against vaccinations for children, specifically the MMR (Measles, Mumps, and Rubella), as the result of fears that these vaccines could cause autism.

Back in 2003, Pew reported that over 80% of Internet users have searched for health information (such as info about fitness or vaccinations) online, and one can imagine that this number has only grown since then, so this example is important in illustrating the potential impact of such health memes. I’ve heard multiple parents mention the supposed vaccination-autism link before when making decisions about whether or not to vaccinate their children, so it’s extremely important to figure out how to make sure these parents get intelligent, credible information when searching on the web. The Murakami paper provides some interesting background information on how this particular meme first started:

In 1997, a group of researchers in the UK lead (sic) by Dr. Andrew Wakefield published a study implying a causal connection between Measles, Mumps, and Rubella (MMR) vaccinations and the development of autism in children. Though further scrutiny of these initial results  disproved the autism-vaccination link – culminating in the withdrawal of endorsements by 10 of the study’s 12 authors – the damage had already been done.

The consequences of this single, spurious, study have already been far-reaching. The resulting backlash precipitated a drop in vaccination rates in the UK (where the study was first published), which has led to an increase in outbreaks of measles over the past decade to the point where measles are once again (after transmission was halted 14 years ago) being considered endemic. Even vaccination rates here in the Bay Area have dropped, with the Examiner reporting that vaccination rates are as low as 50% for some Bay Area schools.

I suppose a lot of this can be chalked up to mainstream media coverage of the original study (as well as coverage of well-meaning, but misguided celebrity activists like Jenny McCarthy). However, a large part of why this meme has continued even after the original study was shown to be dubious is due to social phenomena such as communal reinforcement (or the “millions of people can’t be wrong” phenomenon) that can occur so easily on the web. Because it is so easy to publish information on the web, and because information published tends to persist, it is easy to find a wealth of documents supporting any viewpoint, no matter how much evidence there actually is to support that claim. In this case, one can read 100 different news articles, blog posts, and other online resources based on the Wakefield, et al. study without knowing first that these stories do not corroborate each other (as they are drawn from the same small, possibly falsified study) and second that the original study was actually recently retracted by the journal in the first place.

The way that these stories snowball and take on a life of their own is something that Charles Mackay documented in his book “Extraordinary Popular Delusions and the Madness of the Crowds” (the first lines of which are quoted above). He tackles a variety of subjects ranging from the Dutch tulip craze of the 16th century (cf. “21st century housing bubble“) to alchemy to witch hunts. From this book (written in 1841), we can see that the often reasonable shortcuts that people make when processing new information can sometimes lead to these self-propagating effects which take on a life of their own.

The unfortunate fact is that just because the web gives us access to more information doesn’t guarantee that we are going to choose and use it wisely. This is why building tools to help people make better credibility judgments online is so important, raising two questions:

• How do we extract data from within a single web page to help people make better judgments about the information it contains? I know that there is a good deal of work on this topic in Wikipedia with tools like Wikipedia Scanner and PARC’s WikiDashboard helping to to expose author and change information, but how can we bring tools like these to the web as a whole?
• How do we connect data across web pages to hep propagate changes in information across the web? As an example, if information about the study’s retraction could be propagated to pages reporting on the study, parents reading those pages would be less likely to be led astray, possibly saving lives.

For those who are specifically interested in the MMR vaccine controversy, the Wikipedia page links to a lot of good resources, including a long list of studies conducted in the last decade which show no link between autism and the vaccine.

Given Google’s recent social efforts (Twitter Search, Social Search, Google Buzz, etc.), I am curious to see what they will do with the Aardvark product – will it stand alone as it has or will it find its way into existing or new Google tools? I, for one, would love to see it integrated into Google’s main search. One consequence of Google’s recent launch of Buzz is reminding people that Google has been collecting data on your social network for a while now. If Aardvark were integrated into your Google network, we’d have a out-of-the-box solution for social search (no messy profile-connecting or friend-inviting needed! To me, it seems like one of the biggest hurdles for most people in terms of social search or networking tools is the cost of building up their networks, so this would provide a quick and easy way around that.

What will GoogVark look like? I can’t say I’ve ever used the “I’m Feeling Lucky” button, but I do know that there are times when I can’t quite find the best answers through Google search, and I’d love to be able to seamlessly shift over to social search. I personally would love to see something like this (with an example supplied by Google itself!):

Would this make social search more inviting to you?

P.S. Congratulations to Aardvark’s founders over at The Mechanical Zoo – you guys deserve it!