EVALUATING USER STUDIES IN INFORMATION ACCESS

ORGANISING COMMITEE			PROGRAMME COMMITTEE
Alex Bailey, Canon Technology Europe, UK			Peter Ingwersen, Royal School of Library and Information Science, DK
Ian Ruthven, University of Strathclyde, UK			Kalervo Jarvelin, University of Tampere, FI
Leif Azzopardi, University of Glasgow, UK			Diane Kelly, University of North Carolina, USA
			Iadh Ounis, University of Glasgow, UK
			Simon Sweeney, University of Strathclyde, UK
			Elaine Toms, Dalhousie University, CA
			Ryen White, University of Maryland, US

Workshop is part of the CoLIS5 conference

INTRODUCTION TO WORKSHOP
The complexity in designing, running and analysing a user study is substantially more time consuming and challenging than a simple comparison of empirical measures such as precision and recall. As a result many researchers shy away from the user studies. However, it is only with real user studies that the impact of state of the art research can be truly assessed and the merit of such research validated. As to how a user study should be performed in the context of information access, remains a challenge and those researchers wishing to perform such a study a re faced with many issues to ensure that the research is carried out in an appropriate and unbiased manner.

This workshop aims to:

assess current user based studies in Information Access; discussing the advantages and disadvantages of their methodology,
to provide a forum of discussion for proposed user studies submitted to the workshop, and
to draw up a list of guidelines for future user studies.

For the workshop we solicit two types of contribution: proposed user studies and position papers on user evaluation

CALL FOR POSITION/EXPERIENCE PAPERS
A major aim of the workshop is to initiate discussion on the nature of user evaluation for information access. Consequently, we solicit position papers on the topic of user evaluation and experience papers that outline issues uncovered or methodological innovations tested as part of a user evaluation. Submissions in either category should be between 4-10 pages although experience papers should contain sufficient detail to allow the programme committee to establish the significance of the research described. Papers will be selected for presentation and discussion on the basis of coverage of evaluation issues to ensure that the workshop discusses as many evaluation issues as possible.

CALL FOR PROPOSED USER STUDIES
One of the key activities of the workshop will be to allow researchers to have evaluation proposals critiqued by experts in user evaluations. This represents a unique opportunity to have your proposed user evaluation studied, commented upon and to gain suggestions for evaluating your research before you run your evaluation. As such we ask for submissions to the workshop to be current or proposed user studies (in the area of information access, management, retrieval, visualisation, human computer interaction, etc).

Submission should be approximately 6-10 pages and should include the background and motivation of the study, the research hypotheses, the planned methodology, and the expected outcomes (if possible). Also, any additional information such as planned surveys may be included as supplementary material. Proposals will be selected for presentation and discussion on the basis of coverage of evaluation issues to ensure that the workshop discusses as many evaluation issues as possible.

Submissions of proposals or papers should be emailed to either Alex Bailey (alexb @ cre.canon.co.uk) or Ian Ruthven (ir @ cis.strath.ac.uk) in pdf format. All submissions should be anonymous.

DATES
Submission Deadline: Friday 15th April
Reviewing Deadline: Wednesday 11th May
Notification to Authors: Friday 13th May
Workshop:Wednesday 8th June