Stain guided mean-shift filtering in automatic detection of human tissue nuclei

Background: As a critical technique in a digital pathology laboratory, automatic nuclear detection has been investigated for more than one decade. Conventional methods work on the raw images directly whose color/intensity homogeneity within tissue/cell areas are undermined due to artefacts such as uneven staining, making the subsequent binarization process prone to error. This paper concerns detecting cell nuclei automatically from digital pathology images by enhancing the color homogeneity as a pre-processing step. Methods: Unlike previous watershed based algorithms relying on post-processing of the watershed, we present a new method that incorporates the staining information of pathological slides in the analysis. This pre-processing step strengthens the color homogeneity within the nuclear areas as well as the background areas, while keeping the nuclear edges sharp. Proof of convergence for the proposed algorithm is also provided. After pre-processing, Otsu's threshold is applied to binarize the image, which is further segmented via watershed. To keep a proper compromise between removing overlapping and avoiding over-segmentation, a naive Bayes classifier is designed to refine the splits suggested by the watershed segmentation. Results: The method is validated with 10 sets of 1000 × 1000 pathology images of lymphoma from one digital slide. The mean precision and recall rates are 87% and 91%, corresponding to a mean F-score equal to 89%. Standard deviations for these performance indicators are 5.1%, 1.6% and 3.2% respectively. Conclusion: The precision/recall performance obtained indicates that the proposed method outperforms several other alternatives. In particular, for nuclear detection, stain guided mean-shift (SGMS) is more effective than the direct application of mean-shift in pre-processing. Our experiments also show that pre-processing the digital pathology images with SGMS gives better results than conventional watershed algorithms. Nevertheless, as only one type of tissue is tested in this paper, a further study is planned to enhance the robustness of the algorithm so that other types of tissues/stains can also be processed reliably.