Special Topic Issue on

User-centred Evaluation of Information Retrieval Systems

Guest Editors
Pia Borlund Royal School of Library and Information Science, Aalborg, Denmark.
Ian Ruthven University of Strathclyde, Glasgow, United Kingdom.

Introduction to special issue topic
Evaluation has always been a strong element of Information Retrieval (IR) research, much of our focus being on how we evaluate IR algorithms. As a research field we have benefited greatly from initiatives such as Cranfield, TREC, CLEF and INEX that have added to our knowledge of how to create test collections, the reliability of system-based evaluation criteria and our understanding of how to interpret the results of an algorithmic evaluation. In contrast, evaluations whose main focus is the user experience of searching using IR systems have not yet reached the same level of maturity. Such evaluations are complex to create and assess due to the increased number of variables to incorporate within the study, the lack of standard tools available (for example, test collections) and the difficulty of selecting appropriate evaluation criteria for study.

In spite of the complicated nature of user-centred evaluations, this form of evaluation is necessary to understand the effectiveness of individual IR systems and user search interactions. The growing incorporation of users into the evaluation process reflects the changing nature of IR within society; for example, more and more people have access to IR systems through Internet search engines but have little training or guidance in how to use these systems effectively. Similarly, new types of search system such as recommender systems and interactive IR facilities are becoming available to wide groups of end-users.

This special topic issue will address the current and future position of user-centred evaluation within IR and reflect the increased interest in, use of, and need for user-centred evaluations. The special topic issue will act as a focus for dissemination in best practice in the area of user-centred evaluations by increasing our understanding of appropriate methodologies, our awareness of the effectiveness of evaluation measures and in raising new research directions in the user side of IR.

Submissions for the special topic issue should be high-quality manuscripts on the theoretical, empirical and methodological issues surrounding user-centred evaluations. Submissions that describe evaluations will be welcome as long as they contribute to our understanding of how to perform a user evaluation. Submissions should describe original research and not be under consideration in any other forum.

All submissions should be formatted according to the Information Processing & Management guidelines for papers, references, and citations and should be submitted using the The Elsevier Editorial System for Information Processing and Management When submitting your article please select the appropriate special topic issue as the article type.

Submitted articles will be reviewed according to the Information Processing & Management reviewing criteria and appropriateness to the special topic issue.

We particularly welcome submissions on, but not limited to, the following aspects of user evaluations:

Important dates
Submission of articles: Friday 7th April 2006
Notification to authors: Friday 30th June 2006
Final version of accepted papers: Friday 11th August 2006

All questions regarding submissions should be directed to the special topic issue Guest Editors: Pia Borlund (pb at and Ian Ruthven (ir at