Digital hair segmentation using hybrid convolutional and recurrent neural networks architecture

https://doi.org/10.1016/j.cmpb.2019.05.010Get rights and content

Highlights

  • We propose a deep learning-based model that leverages the hierarchical deep representation learning provided by Convolutional Neural Networks (CNNs) while preserving the spatial dependencies between different parts of the derived feature maps using Recurrent Neural Networks (RNNs).

  • The proposed model was trained using weakly labelled and noisy annotation. Learning from weak labelled has been proven to be an effective method to overcome the lack of annotated data and can achieve good accuracies.

  • The proposed method was tested on real and simulated hair-occluded skin lesion images.

Abstract

Background and Objective

Skin melanoma is one of the major health problems in many countries. Dermatologists usually diagnose melanoma by visual inspection of moles. Digital hair removal can provide a non-invasive way to remove hair and hair-like regions as a pre-processing step for skin lesion images. Hair removal has two main steps: hair segmentation and hair gaps inpainting. However, hair segmentation is a challenging task which requires manual tuning of thresholding parameters. Hard-coded threshold leads to over-segmentation (false positives) which in return changes the textural integrity of lesions and or under-segmentation (false negatives) which leaves hair traces and artefacts which affect subsequent diagnosis. Additionally, dermal hair exhibits different characteristics: thin; overlapping; faded; occluded and overlaid on textured lesions.

Methods

In this presented paper, we proposed a deep learning approach based on a hybrid network of convolutional and recurrent layers for hair segmentation using weakly labelled data. We utilised the deep encoded features for accurate detection and delineation of hair in skin images. The encoded features are then fed into recurrent neural network layers to encode the spatial dependencies between disjointed patches. Experiments are conducted on a publicly available dataset, called “Towards Melanoma Detection: Challenge”. We chose two metrics to evaluate the produced segmentation masks. The first metric is the Jaccard Index which penalises false positives and false negatives. The second metric is the tumour disturb pattern which assesses the overall effect over the lesion texture due to unnecessary inpainting as a result of over segmentation. The qualitative and quantitative evaluations are employed to compare the proposed technique with state-of-the-art methods.

Results

The proposed approach showed superior segmentation accuracy as demonstrated by a Jaccard Index of 77.8% in comparison to a 66.5% reported by the state-of-the-art method. We also achieved tumour disturb pattern as low as 14% compared to 23% for the state-of-the-art method.

Conclusion

The hybrid architecture for segmentation was able to accurately delineate and segment the hair from the background including lesions and the skin using weakly labelled ground truth for training.

Introduction

Skin melanoma is diagnosed using clinical assessment tools to check whether a skin lesion is malignant or not. The assessment is mainly based on the visual inspection of skin lesions. Many clinical tools can help clinicians in melanoma diagnosis. Two of the most commonly used tools are the ABCD rule; and the Glasgow 7-point checklist [1], [2]. The ABCD rule stands for Asymmetry, Border irregularity, Color variegation and Diameter size for the lesion is greater than 6 mm. The Glasgow 7-point checklist has seven criteria: changes in size, shape, colour, the diameter is greater than 7 mm, inflammation, bleeding and localised changes in sensation [2]. The ABCD rule method is one of the easiest methods. Therefore, it has been extensively adopted in clinical practice for the screening of melanoma [1]. There are two types of images for skin lesions: clinical and dermoscopic images. Clinical images are captured using standard cameras or mobile devices. While the dermoscopic images are captured using a dermatoscope to reveal the pigmentation pattern on the skin surface. The dermoscopic images usually have ruler markers and oil stains [3].

Fully-automated melanoma diagnosis using machine learning is an active research area. These techniques have addressed several challenges imposed by a wide variation of the skin lesion regarding colour, texture, shape, size, and spatial location in skin lesion images [5]. However, the challenges imposed by artefacts of skin images are yet to be addressed. These artefacts can be natural such as hairs and veins or external such as bubbles and ruler marks, as shown in Fig. 1. These artefacts may occlude the skin lesions, affecting the ability to identify the nature of the lesion  [6], [7]. However, skin hair has a wide spectrum of variation in characteristics that requires fine tuning by the dermatologist. This, in return, adds more complexity and human source of error to the process. This is why digital hair removal is a necessary preprocessing step before feeding lesion images into an automatic classifier [8], [9]. Digital hair removal shall detect hair and artefacts (e.g. ruler markers) and remove them while preserving the texture of skin image, and most importantly, the lesion. The digital hair removal task is divided into two subtasks, namely hair segmentation and hair inpainting. Therefore, the overall quality of digital hair removal relies on the quality of both subtasks. The available inpainting methods should use surrounding pixels to synthesise the parts of the skin occluded by the hair. There have been great strides in this area, and the latest inpainting methods do provide impressive results especially for synthesising small regions of interest [11], [12]. Thus, the current challenge lies in providing accurate segmentation masks for hair and other artefacts. On the one hand, inpainting an under-segmented hair mask leaves a trace of the hair structure which renders the whole process pointless. On the other hand, inpainting an over-segmented mask forces the inpainting algorithm to synthesise a larger area of the lesion and skin pixels and thus suffer from propagating errors. This, in return, changes the integrity of the lesion texture and consequently affects the automatic diagnosis. This is why the majority of hair removal solutions in the literature favour over-segmentation and invest more in enhancing the quality of inpainting methods. Additionally, there are several types of hair such as thin, thick, entangled, low-contrast and stubble hair. Therefore, relying on a few feature descriptors is a universal solution. Thus, hierarchical representative features are required to account for the subtle differences in the pixel values. This is where deep neural networks architecture provides an automatic way to derive hierarchical feature maps. Deep learning has been successfully used to solve many difficult computer vision problems. Deep learning models are resilient to high variability in skin images. Thus, they can generalise to different hair types and artefacts in images.

In this paper, we address the over-segmentation problem and introduce a novel hair segmentation algorithm based on deep artificial neural networks. The proposed method leverages the hierarchical deep representation learning provided by Convolutional Neural Networks (CNNs) while preserving the spatial dependencies between different parts of the derived feature maps using Recurrent Neural Networks (RNNs). Also, manual hair annotation is a major issue that holds back the development of hair-removal techniques using deep learning. Hair annotation is a time-consuming task, and it is prone to intra-personal variations [13]. To overcome these problems, we trained our model using weakly labelled and noisy annotation. Learning from weak labelled has been proven to be an effective method to overcome the lack of annotated data and can achieve good accuracies [14], [15]. The proposed method is validated using the Jaccard Index to account for false positives which are the main cause of over-segmentation. The segmentation masks produced by the proposed method are then fed into inpainting algorithms [11], [12] to be compared to the state-of-the-art methods on the same dataset [6], [10].

The remainder of the paper is organised as follows. Section 2.1 discusses the literature review of the digital hair removal methods. Then, Section 2.2 introduces the proposed methods for hair segmentation and hair-gaps inpainting. In Section 3, we briefly describe the conducted experiments and the evaluation metrics. The obtained results are reported, followed by a brief discussion in Section 4. Finally, Section 6 presents conclusions drawn from the qualitative and quantitative results.

Section snippets

A Brief Survey on hair detection

In the past few years, many segmentation methods have been developed for digital hair removal. They used adaptive thresholding and threshold-set representation. The input images were converted from RGB colour space to another more representative one. Then, one channel is selected from the chosen colour space. The images were converted into binary images via a given threshold based on luminance values of the selected channel for each image. Potential hair candidates were extracted at each

Experiments

In the proposed work, we conducted four experiments to study the efficacy of the proposed technique:

  • 1.

    Comparative study between proposed method followed by inpainting for detected hair gaps, and three state-of-the-art methods: Dull razor [16], Xie et al. [22], Virtual Shaver [10]. In the proposed method, the segmentation mask was extracted using deep hybrid architecture. Then, the detected hair gaps were repaired by two inpainting methods for comparison: Telea and Bornemann et al. [12].

  • 2.

    Analysis

Quantitative comparison for proposed method

The results demonstrated in Fig. 9 highlight the effectiveness of the proposed solution in detecting thin hair and artefacts in comparison to the state-of-the-art methods Virtual Shaver and DullRazor, respectively. Virtual Shaver tends to produce many false positives as demonstrated by the over-segmentation shown in Fig. 9-d. This leads to unnecessary inpainting which alters the textural integrity of the skin and lesion pixels. On the other hand, masks produced by Dull Razor suffer from false

Discussion

Based on the demonstrated results, the proposed hybrid method can detect the different hair structure based on semantic segmentation without taking into account any prior information about the input images or iterative methods. This proposed method is a robust and efficient framework for digital hair removal that can detect different hair patterns without fine-tuning of parameters and it reduced the false positives. Thus, lesion pixels aren’t altered by unnecessary inpainting.

In this presented

Conclusion

In this presented paper, we proposed an efficient methodology for digital hair removal using the hybrid architecture of neural networks for pixel-wise semantic segmentation. This method was able to segment the hair at different scales using self-learning paradigm with the ability to discard false-positives. Unlike all digital hair removal methods, we were able to utilise all the colour channels for the segmentation process. We trained our model on 2000 images from ISBI 2017, and we used 750

Acknowledgment

This research was fully supported by the Institute for Intelligent Systems Research and Innovation (IISRI) at Deakin University, Australia.

References (61)

  • C. Steger

    An unbiased detector of curvilinear structures

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1998)
  • M. Attia et al.

    Skin melanoma segmentation using recurrent and convolutional neural networks

    Proceedings of the International Symposium on Biomedical Imaging

    (2017)
  • E. Shelhamer et al.

    Fully convolutional networks for semantic segmentation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2017)
  • A. Creswell et al.

    Generative adversarial networks: an overview

    IEEE Signal Process. Mag.

    (2018)
  • W. Stolz et al.

    Abcd rule of dermatoscopy-a new practical method for early recognition of malignant-melanoma

    Eur. J. Dermatol.

    (1994)
  • R. MacKie et al.

    Seven-point checklist for melanoma

    Clin. Exp. Dermatol.

    (1991)
  • R.B. Oliveira et al.

    Computational methods for pigmented skin lesion classification in images: review and future trends

    Neural Comput. Appl.

    (2018)
  • L. Yu et al.

    Automated melanoma recognition in dermoscopy images via very deep residual networks

    IEEE Trans. Med. Imag.

    (2017)
  • J. Burdick et al.

    The impact of segmentation on the accuracy and sensitivity of a melanoma classifier based on skin lesion images

    SIIM 2017 scientific program: Pittsburgh, PA, June 1-June 3, 2017, David L. Lawrence Convention Center

    (2017)
  • J. Koehoorn et al.

    Automated digital hair removal by threshold decomposition and morphological analysis

    International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing

    (2015)
  • A. Telea

    An image inpainting technique based on the fast marching method

    J. Graph.Tool.

    (2004)
  • F. Bornemann et al.

    Fast image inpainting based on coherence transport

    J. Math. Imag. Vis.

    (2007)
  • M. Fiorese et al.

    Virtualshave: automated hair removal from digital dermatoscopic images

    Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE

    (2011)
  • Z. Lu et al.

    Learning from weak and noisy labels for semantic segmentation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2017)
  • H. Zhou et al.

    Feature-preserving artifact removal from dermoscopy images

    Proc. SPIE

    (2008)
  • A. Huang et al.

    A robust hair segmentation and removal approach for clinical images of skin lesions

    Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE

    (2013)
  • M.S.A. Nhi H. Nguyen

    Segmentation of light and dark hair in dermoscopic images: a hybrid approach using a universal kernel

    Proc.SPIE

    (2010)
  • F. Xie et al.

    No-reference hair occlusion assessment for dermoscopy images based on distribution feature

    Comput. Biol. Med.

    (2015)
  • I. Lee et al.

    Hair segmentation using adaptive threshold from edge and branch length measures

    Comput. Biol. Med.

    (2017)
  • M.T.B. Toossi et al.

    An effective hair removal algorithm for dermoscopy images

    Skin Res. Technol.

    (2013)
  • Cited by (24)

    • Does sex matter? Analysis of sex-related differences in the diagnostic performance of a market-approved convolutional neural network for skin cancer detection

      2022, European Journal of Cancer
      Citation Excerpt :

      The most obvious difference between images of males and females in our study was the presence of body hair. It has previously been shown that hair in dermoscopic images might impair physicians' examination [25] and efforts have been made to digitally remove hair from images prior to CNN classification [29]. One publication reported an improved rate of correct CNN classifications after digital hair removal [30].

    • Digital hair removal by deep learning for skin lesion segmentation

      2021, Pattern Recognition
      Citation Excerpt :

      Choosing the best threshold and the best colour space is crucial for the accurate segmentation of the above mentioned methods. Therefore, incorrect choice can lead to high rate of false positive and false negative that can change the texture integrity of the lesions, and leave hair traces and artifacts that affect the subsequent diagnosis [5], as illustrated in Fig. 2. Hence, these methods cannot be applied to most images.

    View all citing articles on Scopus
    View full text