Evaluating Deep Learning Methods for Identifying Nuclei

Caicedo et al. have determined an evaluation framework to assess the improvements in fluorescent images of nucleus segmentation achieved by machine learning versus classical approaches. The team revealed that of the two deep learning modes interrogated (DeepCell and U-Net), there was improved accuracy of nuclei segmentation when both methods were trained with a large fluorescence image set.

deep learningImage Credit: archy13 / Shutterstock.com

A notable development in the field of fluorescent imaging analysis has been made recently with improvements in segmentation. The optimal approach to segmentation, which refers to the process of delimiting the boundaries of objects, is through solutions offered by deep learning solutions.

This subset of machine learning involves artificial neural networks, algorithms inspired by the human brain, that ‘learn’ from repeated interrogation of large quantities of data. A team from X has constructed a framework f evaluation to evaluate improvements in nucleus segmentation seen when adopting deep learning methods.

Traditionally, pixel-overlap is used for nucleus/cell segmentation evaluation, but missing and merged objects are not diagnosed earlier in the segmentation process. The team sought to determine an evaluation method that correctly differentiates between true positives and errors.

Therefore, classical machine learning and image processing algorithms do not satisfactorily capture biologically important error modes. Because of this, cell and nuclei segmentation algorithms are difficult to assess.ow

To compare classical machine learning and image processing algorithms against deep learning algorithms, the team hand-annotated over 20,000 nuclei in a collection of 200 images sampled from differently chemically treated sets.

They used three deep learning strategies to test the accuracy of segmenting each nucleus, each nucleus was labeled as a separate object, and these labels were transformed into masks for background, nucleus interior and boundaries. ‘Masks’ are used to distinguish the area of a cell of interest and exclude other cellular areas of the image that might compromise the analysis. They define a cell compartment, such as the cytoplasm or nucleus.

A convolutional neural network (CNN) was trained using the images and their masks, which generated predictions for pixel classification, where each pixel belongs to one of three classes; background, interior, and boundaries. Each pixel classified as ‘interior’ is assigned a nucleus label.

Two CNNs can be used for nucleus segmentation, DeepCell and U-Net, and these were compared against classic machine learning.

While traditional metrics have focussed on evaluating pixel‐wise segmentation accuracy only, the team used a metric that measures area coverage to identify correctly segmented nuclei. Caicedo et al. also measured other quality metrics, including the number and type of errors.

The team's results showed that deep learning strategies reduce the errors of segmentation and improve segmentation accuracy compared to baseline images obtained from classical machine learning and image processing.  Deep learning methods were reported to ‘excel’ at the correct splitting of adjacent nuclei, being able to recognize boundaries to separate touching nuclei.

Of the two deep learning methods, U-Net was reported to be more sensitive at detecting cell edges and nuclei of all sizes than DeepCell. U-Net did see edges in their absence, however.

The team also determined that more data training was shown to reduce errors in segmentation and improve its accuracy.  They also determined that deep learning required more computing and annotating time compared to classical methods as well as improving the sensitivity of cytometry screens.

Deep learning methods still result in errors spotted by experts, and so require further investigation to quantify these mistakes. Despite this, deep learning offers improved quality, accuracy, and reliability of measurements obtained from the measurements extracted from fluorescence images.

The team concludes that although annotation efforts and computational costs are significant, deep learning methods have an overwhelmingly positive impact on the reliability and quality of fluorescent images.


Funding was provided by the National Institute of General Medical Sciences of the National Institutes of Health. The experiments were run on GPUs donated by NVIDIA Corporation through their GPU Grant Program (to AEC). The team acknowledges support from the HAS‐LENDULET‐BIOMAG and from the European Union and the European Regional Development Funds


Caicedo, J.C. et al. (2019) Evaluation of Deep Learning Strategies for Nucleus Segmentation in Fluorescence Images. Cytometry Part A. DOI: https://doi.org/10.1002/cyto.a.23863

Further Reading

Last Updated: Nov 13, 2019

Hidaya Aliouche

Written by

Hidaya Aliouche

Hidaya is a science communications enthusiast who has recently graduated and is embarking on a career in the science and medical copywriting. She has a B.Sc. in Biochemistry from The University of Manchester. She is passionate about writing and is particularly interested in microbiology, immunology, and biochemistry.


Please use one of the following formats to cite this article in your essay, paper or report:

  • APA

    Aliouche, Hidaya. (2019, November 13). Evaluating Deep Learning Methods for Identifying Nuclei. News-Medical. Retrieved on October 01, 2023 from https://www.news-medical.net/life-sciences/Evaluating-Deep-Learning-Methods-for-Identifying-Nuclei.aspx.

  • MLA

    Aliouche, Hidaya. "Evaluating Deep Learning Methods for Identifying Nuclei". News-Medical. 01 October 2023. <https://www.news-medical.net/life-sciences/Evaluating-Deep-Learning-Methods-for-Identifying-Nuclei.aspx>.

  • Chicago

    Aliouche, Hidaya. "Evaluating Deep Learning Methods for Identifying Nuclei". News-Medical. https://www.news-medical.net/life-sciences/Evaluating-Deep-Learning-Methods-for-Identifying-Nuclei.aspx. (accessed October 01, 2023).

  • Harvard

    Aliouche, Hidaya. 2019. Evaluating Deep Learning Methods for Identifying Nuclei. News-Medical, viewed 01 October 2023, https://www.news-medical.net/life-sciences/Evaluating-Deep-Learning-Methods-for-Identifying-Nuclei.aspx.


The opinions expressed here are the views of the writer and do not necessarily reflect the views and opinions of News Medical.
You might also like...
Utilizing deep learning to enhance the accuracy of mortality forecasts in emergency rooms