Monte Carlo analysis of incomplete paired-comparison experiments

The use of paired-comparison psychophysical experiments is an important technique that is used widely in imaging studies. It is sometimes difficult to compare every stimulus with every other; the number of paired comparisons for n stimuli becomes prohibitive for large values of n. Thus, experiments are often designed by missing some pairs. However, the effect on the accuracy of the estimations of the scale values is not clear. Similarly, if more resources are available, would it be better to recruit more observers making the same paired comparisons or to have the original observers carry out additional paired comparisons? This work seeks to develop a framework for addressing these practical questions surrounding incomplete pairedcomparison experiments design. A Monte-Carlo computational simulation is carried out with an ideal observer model. Results suggest that the proportion of paired comparisons is more critical than the number of observers with small number of stimuli.