Digital hair segmentation using hybrid convolutional and recurrent neural networks architecture
Introduction
Skin melanoma is diagnosed using clinical assessment tools to check whether a skin lesion is malignant or not. The assessment is mainly based on the visual inspection of skin lesions. Many clinical tools can help clinicians in melanoma diagnosis. Two of the most commonly used tools are the ABCD rule; and the Glasgow 7-point checklist [1], [2]. The ABCD rule stands for Asymmetry, Border irregularity, Color variegation and Diameter size for the lesion is greater than 6 mm. The Glasgow 7-point checklist has seven criteria: changes in size, shape, colour, the diameter is greater than 7 mm, inflammation, bleeding and localised changes in sensation [2]. The ABCD rule method is one of the easiest methods. Therefore, it has been extensively adopted in clinical practice for the screening of melanoma [1]. There are two types of images for skin lesions: clinical and dermoscopic images. Clinical images are captured using standard cameras or mobile devices. While the dermoscopic images are captured using a dermatoscope to reveal the pigmentation pattern on the skin surface. The dermoscopic images usually have ruler markers and oil stains [3].
Fully-automated melanoma diagnosis using machine learning is an active research area. These techniques have addressed several challenges imposed by a wide variation of the skin lesion regarding colour, texture, shape, size, and spatial location in skin lesion images [5]. However, the challenges imposed by artefacts of skin images are yet to be addressed. These artefacts can be natural such as hairs and veins or external such as bubbles and ruler marks, as shown in Fig. 1. These artefacts may occlude the skin lesions, affecting the ability to identify the nature of the lesion [6], [7]. However, skin hair has a wide spectrum of variation in characteristics that requires fine tuning by the dermatologist. This, in return, adds more complexity and human source of error to the process. This is why digital hair removal is a necessary preprocessing step before feeding lesion images into an automatic classifier [8], [9]. Digital hair removal shall detect hair and artefacts (e.g. ruler markers) and remove them while preserving the texture of skin image, and most importantly, the lesion. The digital hair removal task is divided into two subtasks, namely hair segmentation and hair inpainting. Therefore, the overall quality of digital hair removal relies on the quality of both subtasks. The available inpainting methods should use surrounding pixels to synthesise the parts of the skin occluded by the hair. There have been great strides in this area, and the latest inpainting methods do provide impressive results especially for synthesising small regions of interest [11], [12]. Thus, the current challenge lies in providing accurate segmentation masks for hair and other artefacts. On the one hand, inpainting an under-segmented hair mask leaves a trace of the hair structure which renders the whole process pointless. On the other hand, inpainting an over-segmented mask forces the inpainting algorithm to synthesise a larger area of the lesion and skin pixels and thus suffer from propagating errors. This, in return, changes the integrity of the lesion texture and consequently affects the automatic diagnosis. This is why the majority of hair removal solutions in the literature favour over-segmentation and invest more in enhancing the quality of inpainting methods. Additionally, there are several types of hair such as thin, thick, entangled, low-contrast and stubble hair. Therefore, relying on a few feature descriptors is a universal solution. Thus, hierarchical representative features are required to account for the subtle differences in the pixel values. This is where deep neural networks architecture provides an automatic way to derive hierarchical feature maps. Deep learning has been successfully used to solve many difficult computer vision problems. Deep learning models are resilient to high variability in skin images. Thus, they can generalise to different hair types and artefacts in images.
In this paper, we address the over-segmentation problem and introduce a novel hair segmentation algorithm based on deep artificial neural networks. The proposed method leverages the hierarchical deep representation learning provided by Convolutional Neural Networks (CNNs) while preserving the spatial dependencies between different parts of the derived feature maps using Recurrent Neural Networks (RNNs). Also, manual hair annotation is a major issue that holds back the development of hair-removal techniques using deep learning. Hair annotation is a time-consuming task, and it is prone to intra-personal variations [13]. To overcome these problems, we trained our model using weakly labelled and noisy annotation. Learning from weak labelled has been proven to be an effective method to overcome the lack of annotated data and can achieve good accuracies [14], [15]. The proposed method is validated using the Jaccard Index to account for false positives which are the main cause of over-segmentation. The segmentation masks produced by the proposed method are then fed into inpainting algorithms [11], [12] to be compared to the state-of-the-art methods on the same dataset [6], [10].
The remainder of the paper is organised as follows. Section 2.1 discusses the literature review of the digital hair removal methods. Then, Section 2.2 introduces the proposed methods for hair segmentation and hair-gaps inpainting. In Section 3, we briefly describe the conducted experiments and the evaluation metrics. The obtained results are reported, followed by a brief discussion in Section 4. Finally, Section 6 presents conclusions drawn from the qualitative and quantitative results.
Section snippets
A Brief Survey on hair detection
In the past few years, many segmentation methods have been developed for digital hair removal. They used adaptive thresholding and threshold-set representation. The input images were converted from RGB colour space to another more representative one. Then, one channel is selected from the chosen colour space. The images were converted into binary images via a given threshold based on luminance values of the selected channel for each image. Potential hair candidates were extracted at each
Experiments
In the proposed work, we conducted four experiments to study the efficacy of the proposed technique:
- 1.
Comparative study between proposed method followed by inpainting for detected hair gaps, and three state-of-the-art methods: Dull razor [16], Xie et al. [22], Virtual Shaver [10]. In the proposed method, the segmentation mask was extracted using deep hybrid architecture. Then, the detected hair gaps were repaired by two inpainting methods for comparison: Telea and Bornemann et al. [12].
- 2.
Analysis
Quantitative comparison for proposed method
The results demonstrated in Fig. 9 highlight the effectiveness of the proposed solution in detecting thin hair and artefacts in comparison to the state-of-the-art methods Virtual Shaver and DullRazor, respectively. Virtual Shaver tends to produce many false positives as demonstrated by the over-segmentation shown in Fig. 9-d. This leads to unnecessary inpainting which alters the textural integrity of the skin and lesion pixels. On the other hand, masks produced by Dull Razor suffer from false
Discussion
Based on the demonstrated results, the proposed hybrid method can detect the different hair structure based on semantic segmentation without taking into account any prior information about the input images or iterative methods. This proposed method is a robust and efficient framework for digital hair removal that can detect different hair patterns without fine-tuning of parameters and it reduced the false positives. Thus, lesion pixels aren’t altered by unnecessary inpainting.
In this presented
Conclusion
In this presented paper, we proposed an efficient methodology for digital hair removal using the hybrid architecture of neural networks for pixel-wise semantic segmentation. This method was able to segment the hair at different scales using self-learning paradigm with the ability to discard false-positives. Unlike all digital hair removal methods, we were able to utilise all the colour channels for the segmentation process. We trained our model on 2000 images from ISBI 2017, and we used 750
Acknowledgment
This research was fully supported by the Institute for Intelligent Systems Research and Innovation (IISRI) at Deakin University, Australia.
References (61)
- et al.
Hair removal methods: a comparative study for dermoscopy images
Biomed. Signal Process. Control
(2011) - et al.
Development of a clinically oriented system for melanoma diagnosis
Pattern Recognit.
(2017) - et al.
Techniques and algorithms for computer aided diagnosis of pigmented skin lesions—a review
Biomed. Signal Process. Control
(2018) - et al.
Hair removal methods: a comparative study for dermoscopy images
Biomed. Signal Process. Control
(2011) - et al.
Beyond pixels: a comprehensive survey from bottom-up to semantic image segmentation and cosegmentation
J. Vis. Commun. Image Represent.
(2016) - et al.
Dullrazor®: a software approach to hair removal from images
Comput. Biol. Med.
(1997) - et al.
E-Shaver: an improved dullrazor® for digitally removing dark and light-colored hairs in dermoscopic images
Comput. Biol. Med.
(2011) - et al.
Towards a computer-aided diagnosis system for pigmented skin lesions
Comput. Med. Imag. Graph.
(2003) - et al.
Hair removal on dermoscopy images
Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE
(2015) - et al.
PDE-based unsupervised repair of hair-occluded information in dermoscopy images of melanoma
Comput. Med. Imag. Graph.
(2009)
An unbiased detector of curvilinear structures
IEEE Trans. Pattern Anal. Mach. Intell.
Skin melanoma segmentation using recurrent and convolutional neural networks
Proceedings of the International Symposium on Biomedical Imaging
Fully convolutional networks for semantic segmentation
IEEE Trans. Pattern Anal. Mach. Intell.
Generative adversarial networks: an overview
IEEE Signal Process. Mag.
Abcd rule of dermatoscopy-a new practical method for early recognition of malignant-melanoma
Eur. J. Dermatol.
Seven-point checklist for melanoma
Clin. Exp. Dermatol.
Computational methods for pigmented skin lesion classification in images: review and future trends
Neural Comput. Appl.
Automated melanoma recognition in dermoscopy images via very deep residual networks
IEEE Trans. Med. Imag.
The impact of segmentation on the accuracy and sensitivity of a melanoma classifier based on skin lesion images
SIIM 2017 scientific program: Pittsburgh, PA, June 1-June 3, 2017, David L. Lawrence Convention Center
Automated digital hair removal by threshold decomposition and morphological analysis
International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing
An image inpainting technique based on the fast marching method
J. Graph.Tool.
Fast image inpainting based on coherence transport
J. Math. Imag. Vis.
Virtualshave: automated hair removal from digital dermatoscopic images
Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE
Learning from weak and noisy labels for semantic segmentation
IEEE Trans. Pattern Anal. Mach. Intell.
Feature-preserving artifact removal from dermoscopy images
Proc. SPIE
A robust hair segmentation and removal approach for clinical images of skin lesions
Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE
Segmentation of light and dark hair in dermoscopic images: a hybrid approach using a universal kernel
Proc.SPIE
No-reference hair occlusion assessment for dermoscopy images based on distribution feature
Comput. Biol. Med.
Hair segmentation using adaptive threshold from edge and branch length measures
Comput. Biol. Med.
An effective hair removal algorithm for dermoscopy images
Skin Res. Technol.
Cited by (24)
HEXA-GAN: Skin lesion image inpainting via hexagonal sampling based generative adversarial network
2024, Biomedical Signal Processing and ControlAutomated hair removal in dermoscopy images using shallow and deep learning neural architectures
2023, Computer Methods and Programs in Biomedicine UpdateA novel minimum generalized cross entropy-based multilevel segmentation technique for the brain MRI/dermoscopic images
2022, Computers in Biology and MedicineDoes sex matter? Analysis of sex-related differences in the diagnostic performance of a market-approved convolutional neural network for skin cancer detection
2022, European Journal of CancerCitation Excerpt :The most obvious difference between images of males and females in our study was the presence of body hair. It has previously been shown that hair in dermoscopic images might impair physicians' examination [25] and efforts have been made to digitally remove hair from images prior to CNN classification [29]. One publication reported an improved rate of correct CNN classifications after digital hair removal [30].
Skin disease diagnosis with deep learning: A review
2021, NeurocomputingDigital hair removal by deep learning for skin lesion segmentation
2021, Pattern RecognitionCitation Excerpt :Choosing the best threshold and the best colour space is crucial for the accurate segmentation of the above mentioned methods. Therefore, incorrect choice can lead to high rate of false positive and false negative that can change the texture integrity of the lesions, and leave hair traces and artifacts that affect the subsequent diagnosis [5], as illustrated in Fig. 2. Hence, these methods cannot be applied to most images.