Machine learning improves cancer cell detection through light scattering

Cytological tests are a common method of screening cancer cells from stained cell samples. Using a microscope, pathologists examine cells collected from bodily fluids, looking for tell-tale signs of malignancy like enlarged nuclei or abnormal cell shapes. Owing to their minimally invasive nature, these tests are widely used for early cancer screening and diagnosis.

In general, getting accurate results from a cytology test depends largely on the skill of the pathologist performing it. However, in some cases, cancerous cells and normal cells can look identical, with differences occurring only at scales smaller than the conventional microscopes can resolve. These often involve changes in nanometric structures, such as actin filaments and microtubules that make up the internal cellular scaffolding, potentially altering how the cell scatters light.

Could machine learning-based systems detect these subtle optical differences?

This research was originally designed by Professor Yoichiroh Hosokawa and Ryohei Yasukuni from Nara Institute of Science and Technology (NAIST), Japan, and Akihiko Ito from Kindai University Faculty of Medicine, Japan. In a recent study published in Scientific Reports on August 3, 2026, the research was led by Assistant Professor Yuka Tsuri from NAIST and set out to answer this question. Their paper was co-authored by Hayata Tsutsui, Wataru Nakata, Risa Onishi, and Mikiya Fujii from NAIST; Fuka Takeuchi and Tomoko Wakasa from Kindai University Faculty of Medicine and Nara Hospital, respectively; Junko Nakamura and Seiichi Hirota from Hyogo Medical University Hospital and School of Medicine; and Ryohei Yasukuni, who is currently affiliated with Osaka Institute of Technology.

The researchers proposed an approach based on dark-field microscopy; a technique that captures the light scattered by an object rather than light absorbed while passing through it. They examined cytology specimens containing cancerous mesothelioma cells and reactive mesothelial cells, which often have very similar appearances, and recorded how they scattered white light with wavelengths between 420 and 720 nanometers. The resulting spectra were fed into a machine learning pipeline that first condensed the data using principal component analysis, then distinguished between cell types with a support vector machine classifier.

After training, the system could differentiate between mesothelioma cells and reactive mesothelial cells with about 91% accuracy in patient-based validation. It also showed potential for distinguishing gastric and urothelial cancer cells, as well as several types of lung and thoracic cancers, although performance varied depending on the tumor types and validation approach. "The result implies that the light scattering spectrum, which contains information at the submicron scale, is very sensitive in detecting differences between cell types compared to visual inspection or image analysis," explains Dr. Tsuri.

Overall, the findings of this study serve as a proof of concept for adding light-scattering information to conventional cytology tests. The research team envisions this technology complementing pathologists' own skill and judgment. "Integrating this spectroscopic system with a microscope could serve as a powerful aid for pathologists and potentially improve the accuracy of their diagnosis," remarks Dr. Tsuri. The researchers are currently working to optimize the optical settings and operation protocol and are applying advanced machine learning methods to further improve diagnostic accuracy.