Poisson distribution. Poisson process. Poisson equation. Poisson kernel. Poisson regression. Poisson summation formula. Poisson’s spot. Poisson’s ratio. Poisson bracket. Euler-Poisson-Darboux equation. This is only a partial list, and yet it shows the degree to which Siméon-Denis Poisson’s work has impacted just about all branches of science. But what is so impressive is not the volume of his contributions but rather their depth, raising a puzzling question: How did Poisson manage to work simultaneously on so many quite different problems and yet stay sufficiently focused to offer deep and lasting contributions?

Well, we had a secret: a notebook and a tiny habit.

Each time Poisson encountered a problem he though fascinating, he would resist the temptation to savor it. He pulled out his notebook instead and made a note of it and promptly returned to the problem that had absorbed him before the interruption. Once he solved the problem at hand, he mulled over the list of problems scribbled in his notebook, then picking as his next challenge the one he found the most interesting.

Poisson’s little secret was lifelong, careful prioritizing.

So, this essentially describes the polar opposite of my work habits, which currently consist of frenetically switching from task to task to ensure that I complete none of them. I’m thinking of giving the priority list a try – has anybody tried a scheme like this and had success with it? Would be curious to hear your story!

The National Science Foundation’s Graduate Research Fellowship Program (GRFP) helps ensure the vitality of the human resource base of science and engineering in the United States and reinforces its diversity.  The program recognizes and supports outstanding graduate students in NSF-supported science, technology, engineering, and mathematics disciplines who are pursuing research-based master’s and doctoral degrees in the U.S. and abroad.  The NSF welcomes applications from all qualified students and strongly encourages under-represented populations, including women, under-represented racial and ethnic minorities, and persons with disabilities, to apply for this fellowship.

In addition to the prestige accompanying the receipt of this fellowship, winners also receive the following:

• Three Years of Support
• $30K Annual Stipend
• $10.5K Cost-of-Education Allowance
• $1K One-Time International Travel Allowance
• TeraGrid Supercomputer Access

The applications first become available in August and are eventually due in early November (at least this was the schedule they followed in 2009-2010). The results were set to be announced in mid-March, though this year they were actually announced in mid-April (I heard that decisions were delayed due to the weather complications in D.C. this winter). According to the site, it looks as if they announced 2000 awardees and 2025 honorable mentions, which seems to be up a great deal from past years.

Why am I writing this post?

For a comprehensive look at applying for NSF (and other similar fellowships), you should check out Philip Guo’s Fellowships Tips page, which is really good and very comprehensive. I personally learn best by example, so my goal in this post is to provide a personal example of the application and review process; hopefully, this will prove helpful to some of you applying in the coming fall.  While I did not win this year, I think it’s helpful to see the essays of others with reviews to get a real sense of what the reviewers are looking for.

If you take one thing away from this post: BE SPECIFIC!

The number one lesson that I gleaned from various sources while applying was “Whatever you write about, BE SPECIFIC”. From my understanding, what you write about and whether or not you actually follow the proposal if you win are both secondary to how specific you can be in your proposal. I wish I could re-find this link now, but when I was applying, I remember reading the application of another student who had posted their application materials and reviews. This student wrote an incredibly detailed proposal for studying land use in Africa; they included specific information about the plots of land they were going to study, the local contacts that they had assembled, even information about satellites from which they were going to pull aerial photos. One of the reviews had a comment along the lines of “Would have appreciated more detail about the study – for instance, what type of analysis are you planning to do on the satellite imagery?” It seems that it is impossible for you to cram too many details into the space provided.

Brief Overview of Written Application Materials:

Essentially, aside from other (important) components such as recommendations, test scores, and the like, the application is comprised of three major written components (see the full list here). The Personal Statement is where you get to talk about your background, your strengths, why you are interested in your research areas, and how winning a fellowship will contribute to your long-term career goals. In the Research Experience Statement, your goal is to essentially discuss why you are qualified to do the work that you are proposing to do. Finally, in the Proposed Plan of Research Statement, you lay out the research question you intend to address, how you are going to answer it, and how that answer will contribute to science and society as a whole.

Throughout these three essays, there are basically two major principles you want to keep in mind:

• Intellectual Merit: How important and original is this research, and how qualified is the applicant to conduct it?
• Broader Impacts: How will this research contribute to science, society, education, underprivileged groups, etc.?

Anyways, with that in mind, here are links to PDF’s of my three statements: Personal Statement, Research Experience, and Proposed Research Plan. Below, I am posting the reviews that I received from my 2 reviewers:

Reviewer 1:

Overall Assessment of Intellectual Merit: Good

The application has a good personal statement and the applicant has a convincing motivation and past research experience background. Nevertheless, I found this application not as competitive because the proposed research application was not specific enough about what exactly it is all about. By the same token it was not stated where the research is. A stronger motivational statement for the proposed research could also strengthen the application.

Overall Assessment of Broader Impacts: Good

The proposed research has an obvious broader impact and benefit to society. Nevertheless, the broader impact statement of this application could be stronger by addressing it specifically and in more detail . Broader impact criteria can be impact on society, integration of research on education, the potential to reach diverse audiences and outreach.

Reviewer 2:

Overall Assessment of Intellectual Merit: Good

Sanjay Kairam has a strong academic record and good research experience. To his credit, he has published several articles in some top-rated conferences. His description of the proposed research is reasonable but I expect a better and more specific explanations of the research plan and methodology.

Overall Assessment of Broader Impacts: Very Good

His interest in social computing stem from his personal experience working in hospital in low income neighborhood. He participated in several activities to promote social computing. The proposed research will have a positive impact on better understanding of social issues.

Conclusions:

In summary, my major takeaway was that while the personal statement and research experience were good, they wanted more details in a few different aspects of my proposed research plan (motivation, methodology, impact).  I was a little puzzled by the phrase “it was not stated where the research is” because I, as is the case with many students, was applying concurrently to graduate schools, so I did not yet know where I would be.

If you are applying for graduate schools, I would advise applying for NSF or another similar fellowship for a number of reasons.  First off, you might get it.  Second, even if you don’t, putting together the application was an incredibly useful exercise towards getting my school applications together – a great portion of my personal and research experience statements found their way into my school application materials.

I hope that getting to see my application statements and the reviews that they earned will help you when you are applying.  If you’d like to know more about my experience applying, talk about your experience, or even have thoughts on what I wrote (hey, I am applying again this year), hit up the comments section. If you are reading this while putting together your application – good luck!

The paper, by Aardvark’s Damon Horowitz and Stanford’s Sep Kamvar, focuses mostly on the architecture of the Aardvark system, from the external representations with which users interact to the internal ranking algorithms on which the system runs. Below, I present a short summary of what they report, focusing on the elements I found most interesting:

The Basic Model: Aardvark’s scoring function is similar to PageRank in that both utilize two primary, but somewhat independently considered components: relevance and quality.

• Relevance in the Aardvark model pertains to the probability that a particular user i can answer the given question q based on the identified topics contained in t.
• Quality in the Aardvark model pertains to the overall probability that a user i can return a satisfactory answer to another user j, regardless of the question.

Indexing Topics: Aardvark computes the relevance score by calculating a distribution of knowledge over topics known by the user using the following sources (keyword-y sounding italicized terms are for convenience only and are not used in the paper):

• Explicit Prompting at sign-up for three “starter” topics about which the user has expertise.
• Social Prompting of a user’s friends to provide topics about which they trust the user’s opinion.
• Structured Parsing of the online profile pages connected to Aardvark by the user (e.g. “Interests” on a Facebook profile).
• Unstructured Parsing of the users’ online homepage, blog, or status updates using a linear SVM to extract overall subject area and a named entity extractor to extract more specific topics.

Indexing Connections: Aardvark computes the quality score by building a set of weighted connections between users using characteristics ranging from social proximity to similarities in demographics or behavior, such as:

• Social Connections either in the form of explicitly defined “friend” connections or implicit “network” connections, such as both being part of the Stanford network.
• Demographic Similarity, which likely includes age, gender, and location based on profile information collected by Aardvark.
• Profile Similarity, which seems to include similar movies and other items which might be listed on other profiles, such as Facebook.
• Vocabulary Match, which they explain with the example of “IM Shortcuts” (i.e. I assume this means it is based on the language you use to interact with Aardvark, but I am unsure.)
• Chattiness and Verbosity Match, which relate to frequency and length of messages used when interacting with Aardvark.
• Politeness Match, which basically seems to mean whether or not say “Thanks!” or not.
• Speed Match, which is a measure of responsiveness to other users.

Analyzing Questions: While all of the other components are pre-computed, this part is computed at question time (obviously). The utilize a number of classifiers to classify the question and then a set of mappers to map the question to a set of topics, noting that “the role of the Question Analyzer…is simply to learn enough about the qeustion that it may be sent to appropriately interested and knowledgeable human answerers”. Here are the classifiers they list (with the names used in the paper):

• NonQuestionClassifier: Determines if input is a valid question.
• InappropriateQuestionClassifier: Determines if input is obscene, spam, or otherwise unsuitable for asking.
• TrivialQuestionClassifier: Determines if input is a simple factual question (examples given: “What time is it now?”, “What is the weather?”). If so, the user gets an automatically generated answer via traditional web search.
• LocationSensitiveClassifier: Determines if the question contains location information; if it does, it passes that information along to the Routing Engine

• KeywordMatchTopicMapper: Checks for string matches against user profile topics (the mapper attempts to classify meaningful vs. spurious matches).
• TaxonomyTopicMapper: Classifies question text using an SVM trained on an “annotated corpus of several million questions” (where did they find that?)
• SalientTermTopicMapper: Extracts salient phrases using a noun-phrase chunker and tf-idf and finds “semantically similar user topics”.
• UserTagTopicMapper:Utilizes tags explicitly provided by the asker or other answerers and maps them to user topics.

This description of the routing algorithm comprises the main function of the paper. After some more description of how users interact with the system, the authors provide some interesting data collected over the past several months of use (from the beta launch in March 2009 until October 2009).  Here’s a quick run-down of the more interesting facts that they presented:

• Strong User Growth: As of October 2009, they reported 90,361 user accounts, and users appear to be remaining active (in the study period, over 1/2 the users actively generated content and over 2/3 of the users passively participated).

User Growth on Aardvark (graph taken from the paper).

• Higher Query Contextualization: Aardvark queries average 18.6 words in length while the average query length reported for web search is between 2.2 and 2.9 words (citing previous comparison and characterization studies).  They further state that “98.1% of questions are unique”, though I am unsure as to how exact they are being about matching (I am sure the question “What’s a great restaurant in SF” has been asked 1000 times in different forms). In addition, they report from manual scoring of 1000 randomly selected questions that 64.7% of questions asked have a subjective element, with advice about travel, restaurants, and products being specifically popular.
• Fast, High-Quality Answers: They report that 87.7% of questions get answers and 57.2% received an answer within 10 minutes. They report that 70.4% of answers receiving feedback are rated as “good” and only 15.5% are rated as “bad”. Interestingly, they observe a notable difference in feedback on answers from users within the asker’s social network (76% rated as food) and outside the asker’s network (68% rated as good).

Questions on Aardvark (chart taken from the paper).

Overall, I really enjoyed reading this paper. After using Aardvark for over a year now, it was really interesting to get to peer inside and see how the system works, and a lot of great details were provided about the ranking engine.

One place where I feel that the authors missed the mark was in the cursory side-by-side evaluation which pitted Aardvark against Google for a set of 200 questions randomly selected from the Aardvark system. They report that 71.5% of the questions studied were answered successfully on Aardvark, while 70.5% of the questions were answered successfully on Google. This comparison seems mostly useless as the questions, having been pulled from the Aardvark system in the first place, are ones that were specifically chosen because they are better adapted to what is being called ‘social search’. This comparison left me desirous of more investigation into two main questions.

“What makes a search engine ‘social’ in the first place?”

The distinction between social and non-social is extremely murky, something Brynn and I discovered when working on our Social Search paper. It has been argued before (one small example here) that Google’s PageRank algorithm is inherently social, as it aggregates information provided by people (links to one another) to rank results. However, it is clear that something seems categorically different between Google and what people perceive to be ‘social search’. When it comes down to it, even though everyone is excited about Google’s forays into “Social Search”, there’s nothing all that fundamentally different about Google indexing your blog and your tweets than any other documents extant on the web.

To me, it seems that the key difference is really the change in the direction of interaction. While Google takes a query (question) and compares it against traces of discussion about that question from the past (web documents), systems perceived as ‘social’ take a question and attempt to generate new answers in the future. This change in direction is what allows for the higher context that makes ‘social’ search answers so much more rich (at least for some questions.)  Perhaps we need a different word to define this phenomenon – ‘real-time search’ seems to get at it more, but has its own problems.  Perhaps something like ‘generative search’? I really don’t know.

“Why do we need a social search engine at all?”

This one seems like the best fodder for a follow-up study by Aardvark. While they do provide a rough breakdown of the types of questions asked on Aardvark (see pie chart above), I think that a comparison might have been much more interesting if they had looked at a variety of classes of user needs and had compared the relative efficacy of searching on Aardvark and a traditional search engine such as Google. It is clear that ‘social’ will work much better for some needs and much worse for others, but up to this point, people who talk about social search always seem to use the same types of examples (travel, restaurants, and products, for instance). It would be great to get a clear idea over a wide range of needs and use cases where systems such as Aardvark can provide benefits over existing tools.

Anyways, for those of you interested in ‘social search’ and search systems, I encourage you to read this paper and tell me your thoughts!

Upon reading this study, I noticed two things:

1) The original experiment solicited participants by offering to donate $10 to charity for their time; resulting in a net cost of at least $14K, raising the second question: Given the growth of sites like Mechanical Turk, can we get comparable results for less money?

2) The data was originally collected in December 2009 1999, which is almost exactly 10 years ago. Content on the web and our interactions with it have changed a great deal since then – I mean, in 1999, Google had only existed for a year, blogs were still relatively uncommon (LiveJournal had just started), and Wikipedia didn’t even exist yet! This raised the question: Given that the average Internet user now has a far greater amount of experience navigating the Web, should we expect the responses to be different 10 years later?

I decided to explore both of these questions through a survey on Mechanical Turk; in December 2009, I posted a HIT to Mechanical Turk replicating the original study as closely as possible. The 51 statements about web site elements were available from Fogg’s original paper, the same 7-point Likert scale was used, and the survey items were randomized.  I paid $0.05 per HIT, and I ended up getting 327 responses, with none thrown out due to quality.

An initial, high-level examination of the responses showed that they actually matched the 1999 data fairly well.  Average Likert ratings for each item correlated highly with average ratings for the same items in the 1999 data with R^2 = 0.96.  One difference in the data was that answers were compressed (closer to 0) overall, so for the purposes of comparison, I transformed the 2009 data using the transformation (1.1677X – 0.0003) to match the 1999 data (note that this has no effect on the correlation coefficient).

The first analysis in Fogg, et al. was to separate the various elements into 7 “scales” using factor analysis (Real-World Feel, Ease of Use, Expertise, Trustworthiness, Tailoring, Commercial Implications, and Amateurism).  Below, I present comparisons for items in each scale.  I highlighted items that deviated from the 1999 values by more than 0.25 (without the original data, I couldn’t do much more in-depth comparison), but this might give some rough idea of which elements are more important now than they were in 1999 (red) and which elements are less important (purple):

Elements related to the "Real-World" feel scale are rated similarly from 1999 to 2009.

One difference is that people seem more critical of long download times than in the past.

For some reason, it seems that displaying an award helps your site's credibility more than 10 years ago, and that providing a lot of news stories matters less.

Stating your policy on content and ending in ".org" are more important to people now - could this be a cultural shift in response to sites like Wikipedia? Overall, it seems as if links out to other sites matter less for credibility now.

Email confirmations are more important now than in 1999.

Outside advertisements and an e-commerce focus matter more for credibility now than in the past. People are paying less attention to the commercial purpose of sites, as well as the number of ads and their integration with content (Google is a favorite site for many, after all.)

People seem to notice domain name mismatches now (more public knowledge of phishing/identity theft now?), but less attention to multi-lingual sites.

People seem more willing to trust free financial sites now than in the past.

Anyways, as we might have guessed, the answers from 2009 seem to match the answers from 1999 pretty well.  The elements that made things highly credible or highly ‘un-credible’ in the past seem to have remained constant, and those which didn’t matter then seem not to matter too much now.  Some interesting elements are noted in the captions with some rough conjectures as to why some of them might be trending the way they are.

The rest of the Fogg paper focused on characterizing differences in scale responses due to demographic differences between participants, but I found that part of the study less convincing as they averaged over the positively and negatively phrased elements on each scale (which I think makes the interpretation somewhat confusing).

Now, back to our two questions:

1) It looks as if using Mechanical Turk, we were able to get reasonable answers that fairly closely matched those from the original study.  Total price tag: 327 response * $0.05 = $16.35 paid to participants (plus Amazon’s cut), which made this study about $13,980 cheaper than the original one.

2) We see above a few small changes in what makes web sites credible, but overall, people are looking at the same things, meaning that we should continue taking the same factors into account when designing websites. I’ve made some guesses as to why these elements may have changed over time, but I’m curious to hear what you think, so leave a comment!