Skip to content
Tags

, , , , , , ,

Meta-Review: The Role of Domain Expertise in Web Search

by skairam on October 27th, 2009

This is a first post in a new format that I’m trying out: the “Meta-Review”.  Besides the fact that it starts with an “M” (thus fitting with my category naming format), I’m calling it a “Meta-Review” because it’s composed of notes and thoughts about a handful of papers all mashed together.  This isn’t intended to be a carefully thought-out treatise on the papers discussed, but instead is really just a more public version of my immediate thoughts and notes (if I’m going to write them down anyways, why not share?)  Comments, discussion, and pointers to additional/related papers are encouraged, as they would benefit other readers (and more importantly, me).

Here’s a quick list of the papers mentioned:

  • “How Medical Expertise Influences Web Search Interaction” [1] and “Characterizing the Influence of Domain Expertise on Web Search Behavior” [2] by Ryen White, Sue Dumais, and Jaime Teevan.  This poster abstract and longer paper present a large-scale, log-based analysis of web searches in 4 domains (Medicine, Finance, Law, and Computer Science), looking specifically at how domain experts differ from non-domain experts in terms of search behavior.  The data for the study were extensive, comprised of a sample of URL visits from users of a browser toolbar over the course of a 3-month period and representing “more than 10 billion URL visits from more than 500 thousand unique users.”
  • “Knowledge in the Head and on the Web: Using Topic Expertise to Aid Search” [3] by Geoffrey Duggan and Stephen Payne.  This paper looks at the role of domain expertise in predicting search performance for people searching within their domain of expertise.  The study involved asking 34 university students trivia questions on two topics – Football (they meant to write “Soccer”) and Pop Music – and asking them to answer, first using only their own knowledge, and then again with the help of the Internet.
  • “Web search behavior of Internet experts and newbies” [4] by Christoph Holscher and Gerhard Strube.  This is a somewhat earlier paper focusing on identifying the search strategies of internet (search) experts, and then using that ifnromation to help compare the effects of serach expertise and domain expertise on search performance.  In the first study, they had 12 internet experts first do a mental walk-through of their search strategies and then carry out real search tasks using a teach-aloud/think-aloud sort of protocol.  In the second study, they had 24 university students conduct web-based search tasks pertaining to economics – they were divided by domain expertise (half were economics students) and search expertise (assessed by interview and pre-test).
  • “Domain knowledge, search behaviour, and search effectiveness of engineering and science students: an exploratory study” [5] by Xiangmin Zhang, Hermina G.B. Anghelescu, Xiaojun Yuan.  This paper examined the relationships connecting domain knwoledge, search behavior, and search effectiveness.  The study established the domain knowledge of 22 engineering studies through familiarity with terms from an engineering thesaurus, and then had them search on 3 assigned topics.

The papers that focused on quantifying search behavior showed that domain experts tended to do more exploration overall than domain novices.  [2] found that they issued more queries, they branched more (branching defined as stepping back to a previous page and then moving forward to a new page), they visited a larger number of unique domains, and they spent a longer time overall per search tasks.  [5] also found that domain experts tended to issue more queries (34.64 queries/subject vs. 20.09) over the course of their search tasks.  In addition, the domain experts explored this space faster: [3] found that greater topic expertise led to less time spent per page and to faster decisions about whether or not to stop a line of inquiry, a finding corroborated by [4].

One interesting point of disagreement involved the length of queries.  [2] pointed to past literature demonstrating that domain experts tended to issue longer queries and more technical query terms, a result replicated in their study.  Longer queries were seen from domain experts in [5], as well (4 terms/query vs. 2.86).  [3], however, found that the domain experts studied used shorter queries than the domain novices, contradicting these other studies (though the scope of consideration was restricted greatly to come to this conclusion – they looked at just 2 of the football questions).  In the first study from [4], when the search behavior of the internet experts was compared against search logs from the Fireball Search Engine, it was found that the internet experts use longer queries (3.64 vs. 1.66 words), but in the second study, those with domain knowledge were found to make shorter queries than those without (1.97 vs. 2.96 words).

The papers that attempted to highlight specific search strategies also revealed some interesting differences.  [2] examined the domain suffixes of the sites visited, and noticed that domain experts tend to visit different types of sites than domain novices.  Computer Science experts, for instance, were more likely to visit *.org or *.edu sites than novices, while novices were more likely to visit *.com sites, representing that experts might be more familiar with academic or industry sites, while novices might be more familiar with consumer-oriented commercial sites.  [4] also found that so-called “double experts” (those with Internet AND domain expertise) tended to navigate directly to “go-to” sources of information, while all other groups were more likely to start with search engines.

Regarding overall performance, it is perhaps not surprising that domain experts performed better in all search tasks.  [3] attempts to distinguish between searching within one’s domain for information already known vs. searching within one’s domain for information that one doesn’t already know, and found that domain experts perform better in both scenarios.  Because [2] did not control user tasks, they coded successes as logged searches where the final click was a URL and failures as when the final click was another search.  Given this coding, they found that experts were more successful than novices when searching in-domain, but that these same experts performed the same as novices when searching for information out of their domain of expertise, highlighting the difference between domain expertise and search expertise.

Some of the interesting questions that come out of this field of research relate to how we can transfer the advantage that domain (and search) experts have to novices.  One possible method is to pin down what these experts are doing that helped them perform better and attempt to work these strategies into instruction.  Obviously, Internet search skills are already immensely important, and I would hope that this would trickle down into educational curricula (if they haven’t already).

Domain experts find information faster because their expertise in the space allows them to identify relevant information faster and to build off of it.  But, for those of us who are attempting to use the web to learn things on our own, there is a serious boot-strapping problem here.  As someone who is mostly self-taught when it comes to programming, I know how difficult it is to face the problem of searching for information when you are not entirely sure what to search for.  Once you get over the initial learning curve, it becomes much easier.  For those of us interested in building new technologies, here is a challenge: How can we create tools that support domain novices by doing this bootstrapping for them?

If we can find a way to identify domain novices and present them with useful information such as definitions or important “go-to” sources, we can significantly speed up their learning so that they can more quickly fend for themselves.  Faceted search tools such as MrTaggy take a solid step towards tackling this problem by providing searchers with additional cues that provide context.  More detailed study is needed regarding how to connect the gaps in which domain novices get lost – as better tools become available for providing socially or computationally-derived contextual data, it will be interesting to see what technologies evolve to support these needs.

Links for Papers Above (Most from ACM Digital Library):

Some Other Reading on this Topic:

  • “The Effects of Topic Familiarity on Information Search” by Diane Kelly, Colleen Cool in Joint Conference on Digital Libraries (JCDL) 2002.
  • “Domain-Specific Search Strategies for the Effective Retrieval of Healthcare and Shopping Information” by Suresh K. Bhavnani in Conference on Human Factors in Computing Systems (CHI) 2002.
  • “The Effects of Domain Knowledge on Search Tactic Formulation” by Barbara M. Wildermuth in Journal of the American Society for Information Science ant Technology, 2004.

From → /Metareview

Leave a Reply

Note: XHTML is allowed. Your email address will never be published.

Subscribe to this comment feed via RSS