MSIM: A change detection framework for damage assessment in natural disasters

https://doi.org/10.1016/j.eswa.2017.12.038Get rights and content

Highlights

  • A new social image model by exploiting tags and locations is proposed.

  • Image pair identification can be enhanced by clustering and hash techniques.

  • Proposed relative position annulus model is robust to boundary rotation.

  • Proposed change detection method outperforms the existing techniques.

Abstract

This paper investigates how social images and change detection techniques can be applied to identify the damage caused by natural disasters for disaster assessment. We propose a multi-step image matching-based framework (MSIM), which takes advantages of fast clustering, near duplicate image detection and robust boundary matching for the change detection in disasters. We first model the social images by exploiting their tags and locations. After that, we propose a recursive 2-means algorithm over the new data model, and refine the near duplicate detection by local interest point-based matching over image pairs in neighboring clusters. Finally, we propose a novel boundary representation model called relative position annulus (RPA), which is robust to boundary rotation, location shift and editing operations. A new RPA matching method is proposed by extending dynamic time wrapping (DTW) from time to position annulus. We have done extensive experiments to evaluate the high effectiveness and efficiency of our approach.

Introduction

Before, during and after the natural disasters, information about the damages and the current situations are vital for people to make decisions for their next actions. For example, the Nepal earthquake in 2015 did a huge damage to everything, including buildings, roads, infrastructure, and people, which resulted in 8,019 people died and 17,866 people injured; in a Tokyo earthquake, people are still able to walk back home since the earthquake damage the infrastructure, but not the buildings and paths. A recent study (Hiroi, Sekiya, Nakajima, Waragai, & Hanahara, 2011) found that people would like to know earthquake size and epicentre. Moreover, knowing these information could prevent the secondary and tertiary disasters. It is also found that in Tokyo, only 67.8% of people managed to get back their house on the day of an earthquake, and the rest 32.2% had to become “homeless”, among them 2% failed to going back home only because they could not find the safe path. People need the information about earthquakes, but there is always the question “How could the people get the information of earthquakes?”.

Recently, many researchers had approached to detect the damages caused by natural disasters by using the change detection techniques with the aerial images. However, the aerial images consume longer time to retrieve and harder to get compare to the other images. On the other hand, social media is pervasive, and updates very quickly especially on large events, e.g., natural disasters, by millions of people all the time. Thus, we investigate the problem of change detection from social images, so as to let the public aware of the latest situations of natural disasters on the spot. One of the challenges here is the large-scale of social images, which makes current change detection techniques infeasible if not impossible. The limitation of using the large-scale images with current change detection techniques is time cost. As existing techniques detect changes based on pixel level comparison (İlsever & Unsalan, 2012) without index support or query optimization or both, the time cost for image comparison is high. When applying them to large scale social images, the efficiency issue becomes even unacceptable. The second challenge is the unavailability of some special features, like building shady, used in traditional change detection (Turker & Sumer, 2008). In shared communities, most social images do not have shady because of low shooting angles, thus the shady-based matching cannot be conducted. Forcing the existing techniques on the social images will cause low detection quality. Finally, traditional change detection for aerial images suppose the image pairs are known to the same location points, which is not true in media shared communities.

To address these challenges, we have proposed a basic solution (Kito et al., 2017) for effective and efficient change detection from social community. Each image is represented by its local interest descriptors called Principal Component Analysis-Scale Invariant Feature Transform (PCA-SIFT). Then each image pair from the same source is identified by one-to-one matching over their PCA-SIFT descriptors. After that, boundaries of each image are modeled as a number of Relative Position Annulus (RPA) representations, which shows the difference of neighbouring edge lengths and is robust to the view point rotation and other global transformations of the same objects in images. Finally, the changes of each image pair during disasters are detected by proposing an extended dynamic time wrapping metric over the RPA representations. In this paper, we extend our previous work with a novel semantic data model over locations and tags, and propose a hash-based clustering method to further improve the efficiency when identifying each image pair from the same source, so the whole process of change detection can be conducted in real time. Specifically, we first represent the metadata of each image as a set of weighted tags, and propose a Social Image Similarity function (SIS) over image metadata and locations. Then, we extend the recursive 2-means clustering algorithm (Shen, Ooi, & Zhou, 2005) from the vector space under Lpnorm to the key word set space, so the Jaccard-based SIS measure can be applied. A hash-based technique is applied to improve the efficiency of clustering process. By deploying this extended recursive 2-means clustering over the whole image dataset, a number of small clusters are generated. Images in the same cluster or neighbouring clusters will have the high probability of being the image pairs of the sources. Following that, we conduct PCA-SIFT based matching, which determines whether two images are really referring to the same source. We have conducted experimental evaluation over two large real datasets to demonstrate the effectiveness and efficiency of our framework. To summarize, the contributions of this paper are as follows:

  • We propose a new social image model by exploiting tags and locations and a new similarity function SIS over it. The new model can effectively and efficiently capture the similarity between images in terms of semantics.

  • We propose an SIS-based clustering method by extending the recursive 2-means algorithm, then deploy a PCA-SIFT-based matching over images for finding the images pairs of the same source. With clustering, the copies can be detected by only checking the images in neighbouring clusters.

  • We use djb2 hashing techniques to reduce one by one tag comparisons, so the efficiency of whole process of SIS-based clustering is greatly improved.

  • We conduct extensive experiments on two large real image datasets to verify the performance of our proposed solution in terms of effectiveness and efficiency.

The rest of the paper is structured as follows: Section 2 reviews the related work; Section 3 presents our change detection framework; Section 4 details the social image modelling; Section 5 presents the proposed change detection method; Section 6 includes the experiment evaluation; Section 7 concludes the paper.

Section snippets

Related work

In this section, we review the existing research closely related to this work, including the image copy detection and change detection.

Framework

In this section, we first define two terms, change and change detection, and then present the overview of the change detection framework for social images.

Definition 1

In image processing, change is defined as the difference between two pixels or the objects in different images. The difference varies in different situations. In this paper, the difference is limited to the damage to the roads, the buildings or the infrastructures, which are caused by natural disasters. For instance, when a bridge breaks

Data modelling

This section presents how to model images by tags and uploading location information. Intuitively, while tags describe the content information of images conceptually, the location information is the key feature to evaluate whether two different images come from the same place. When images are uploaded in social communities such as Flickr, users attach tags and locations with them. We collect tags and location to model social images. The missing tags can be supplemented by tag recommendation

Change detection over large data sets

This section presents our change detection method, including how to identify the image pairs from the same source and how to detect the changes in images.

Experiment

This section examines the effectiveness and efficiency of the proposed method.

Conclusion

In this paper, we study the problem of change detection from media sharing communities for damage assessment in natural disasters. First, we propose a new concept-level image data model over tags and locations, and extend recursive 2-means clustering to our model for fast detection of images from the same sources. Then, we propose a robust boundary representation model together with the matching over it for effective change detection for damage assessment. Finally, we have conducted extensive

References (29)

  • P.K. Pearson

    Fast hashing of variable-length text strings

    Communications of the ACM

    (1990)
  • L. Al Shalabi et al.

    Data mining: A preprocessing engine

    Journal of Computer Science

    (2006)
  • Appleby, A. (2008). Murmurhash...
  • H. Bay et al.

    Surf: Speeded up robust features

    Computer vision–ECCV

    (2006)
  • D.J. Berndt et al.

    Using dynamic time warping to find patterns in time series

    International conference on knowledge discovery and data mining

    (1994)
  • F. Chierichetti et al.

    Finding the Jaccard median

    Proceedings of the twenty-first annual ACM-SIAM symposium on discrete algorithms

    (2010)
  • L. Gong et al.

    Earthquake-induced building damage detection with post-event sub-meter vhr terrasar-x staring spotlight imagery

    Remote Sensing

    (2016)
  • U. Hiroi et al.

    Questionnaire survey concerning stranded commuters in metropolitan area in the Great East Japan Earthquake

    The Journal of Social Safety Science

    (2011)
  • M. İlsever et al.

    Two-dimensional change detection methods: Remote sensing applications

    SpringerBriefs in computer science

    (2012)
  • D.U. Ismail Bin Mohamad

    Standardization and its effects on k-means clustering algorithm

    Research Journal of Applied Sciences, Engineering and Technology

    (2013)
  • L. Juan et al.

    A comparison of sift, pca-sift and surf

    International Journal of Image Processing

    (2009)
  • Y. Ke et al.

    Pca-sift: A more distinctive representation for local image descriptors

    IEEE Computer Society Conference on Computer Vision and Pattern Recognition

    (2004)
  • N. Kito et al.

    Change detection from media sharing community

    Asia-Pacific web (APWeb) and web-age information management joint conference on web and big data

    (2017)
  • H. Lejsek et al.

    Scalability of local image descriptors: A comparative study

    ACM international conference on multimedia

    (2006)
  • Cited by (18)

    • Superpixel-based multiobjective change detection based on self-adaptive neighborhood-based binary differential evolution

      2023, Expert Systems with Applications
      Citation Excerpt :

      In the field of remote sensing (RS) (Kurban, 2022; Zhang & Zhu, 2011), it obtains the RS images of the same area in different periods and analyzes the remote sensing image information so as to detect changes of the land surface of the geographical region. At present, it has attracted extensive interest due to a great quantity of applications in diverse fields such as disaster assessment (Qin, et al., 2018), land use monitoring (Gupta et al., 2019), urban expansion monitoring (Bouziani et al., 2010; Tison et al., 2004) and so on. As a common category of RS images, SAR images have strong penetration ability and accurate azimuth resolution as well as range resolution (Dong, et al., 2021).

    • Large-scale deep learning based binary and semantic change detection in ultra high resolution remote sensing imagery: From benchmark datasets to urban application

      2022, ISPRS Journal of Photogrammetry and Remote Sensing
      Citation Excerpt :

      The purpose of change detection is to discover land-cover changes and interpret the process of feature change, where the results determine which objects have changed, both when and where. This information can provide help for urban construction (Huang et al., 2017; Daudt et al., 2018b; Amini Amirkolaee and Arefi, 2019; Tian et al., 2021), disaster emergency response (Vetrivel et al., 2018; Qin et al., 2018; Liu et al., 2019a; Zhang et al., 2020b), environmental protection (Bindschadler et al., 2010; Tian et al., 2014; Zhu et al., 2019; Ochtyra et al., 2020), and other applications (Jin et al., 2017; Hu et al., 2018; Maiya and Babu, 2018; Liu et al., 2019b), thereby assisting with decision-making. As the process of urbanization has accelerated in recent years, how to better detect urban changes has become a research focus.

    • Accelerated genetic algorithm based on search-space decomposition for change detection in remote sensing images

      2019, Applied Soft Computing Journal
      Citation Excerpt :

      Change detection is one of the most important research topics in remote sensing image processing, and has been widely used in environmental monitoring [1], natural disaster damage assessment [2], forest resources monitoring [3], agricultural surveys [4], and so on.

    View all citing articles on Scopus
    View full text