Is it possible to measure the quality of a scientist?

In the past years it has become very popular to use metric data to evaluate the quality of science. England, for instance, has for several years used a complicated RAE (Research Assessment Exercise) system, where an expert panel evaluate the quality of a research institution (typically on the department level) based on the number of publications, citations and the quality of the publications.

However, there is also a need to evaluate the individual scientists when hiring or for fellowship and scholarship applications. Traditionally this has been done by individually evaluating the scientist’s achievements (although with the number of publications being used as an indicator of quality), but now in certain places, especially at some American universities, it is becoming popular to metric data.

One of the most popular metric methods is the so-called h-index, which provide information on both the number and the quality of the scientist’s publications.

The h-index is found by ranking all publications after the number of citations they have received. When the publication-number and the number of citations match this number gives the h-index. If a young scientist for instance has 10 publications with the following number of citations 0, 5, 17, 1, 2, 0, 10, 8, 3, 7. Ranking them gives (1) 17, (2) 10, (3) 8, (4) 7, (5) 5, (6) 3, (7) 2, (8) 1, (9) 0, (10) 0. Our young scientist thus has a h-index of 5.

However, there are several problems with the h-index. It is for instance almost impossible to compare values across scientific disciplines, since the productivity in form of published papers varies significantly.

The h-index does, furthermore, not discern whether the scientist is a single author in the paper or whether he is just one author among many others. The quality and reputation of the journal in which the paper is published is also not included.

Finally, the h-index discriminate against younger scientists, since they usually have only few published papers and their papers has been published recently thus not giving them much time to accumulate citations.

Variations of the h-index exist which attempt to correct for these omissions, but nonetheless you cannot (and should not) evaluate the quality of a scientist solely from one metric value.