Integrating joint feature selection into subspace learning: A formulation of 2DPCA for outliers robust feature selection

Abstract

Since the principal component analysis and its variants are sensitive to outliers that affect their performance and applicability in real world, several variants have been proposed to improve the robustness. However, most of the existing methods are still sensitive to outliers and are unable to select useful features. To overcome the issue of sensitivity of PCA against outliers, in this paper, we introduce two-dimensional outliers-robust principal component analysis (ORPCA) by imposing the joint constraints on the objective function. ORPCA relaxes the orthogonal constraints and penalizes the regression coefficient, thus, it selects important features and ignores the same features that exist in other principal components. It is commonly known that square Frobenius norm is sensitive to outliers. To overcome this issue, we have devised an alternative way to derive objective function. Experimental results on four publicly available benchmark datasets show the effectiveness of joint feature selection and provide better performance as compared to state-of-the-art dimensionality-reduction methods.

Introduction

With the recent advancement in data acquisition devices, acquiring data at faster rates and increased resolution has improved substantially over recent years. The data interpretation process, however, is facing several challenges due to high dimensionality. Not only for the classification, dimensionality reduction is also a serious challenge for several other domains such as data visualization, data compression, pattern recognition, and computer vision. The aim of dimensionality reduction is to transform the high-dimensional data into low-dimensional representation by preserving the quality of the data so that it could be classified efficiently. To deal with this issue, several vector-based methods are in use during the last two decades such as Principal Component Analysis (PCA) (Turk & Pentland, 1991), Linear Discriminant Analysis (LDA) (Belhumeur et al., 1997, Razzak et al., 2010, Ye et al., 2018), LPP (He & Niyogi, 2004), SPP (Qiao, Chen, & Tan, 2010), SPPE (Zhang, Yan, & Zhao, 2013), Isomap (Zhang et al., 2018) and NPE (He & Niyogi, 2004). Principal Component Analysis is one of the extensively used unsupervised dimensionality reduction method that projects high-dimensional representation into linear orthogonal space. However, one of the major drawbacks is that PCA is linear combination and loading are non-zero. This makes PCA data interpretation difficult, and it is still sensitive to outliers (as its covariance matrix is derived from ${\ell }_2$-norm that affects its performance. Thus, it fails to deal with outliers that often appears in real-world data. Moreover, before applying PCA and LDA, there is need to convert the image into one-dimensional vector, thus it may not exploit image’s spatial structural information very well (Feng et al., 2013, He and Niyogi, 2004, Netrapalli et al., 2014, Turk and Pentland, 1991, Vaswani et al., 2018, Xu et al., 2010, Yi et al., 2017, Zou et al., 2006) which is very important for image representation. To overcome these issues, several variants of PCA have been proposed to improve the effectiveness of dimensionality reduction and robustness against outliers.

Matrix-based subspace learning methods have been widely applied for dimensionality reduction (Li et al., 2017, Li et al., 2010, Tian et al., 2017, Yang et al., 2004, Yang et al., 2005). Results showed that 2DPCA (Yang et al., 2004), 2DLDA (Yang et al., 2005), multi-linear PCA (Lu, Plataniotis, & Venetsanopoulos, 2008), and JGSPCA (Khan, Shafait, & Mian, 2015) are far more efficient as compared to one-dimensional subspace learning, due to its direct formulation based on two-dimensional images. Two-dimensional subspace learning methods directly calculate the class scatter metrics from images, hence can reveal the spatial structural information of image that is quite important for image classification task. To select important features, several efforts have been made such as robust 2DPCA, utilization of nuclear norm, ${\ell }_1$, ${\ell }_{2,1}$, and Frobenius-norm that showed considerable improvement against outliers and able to select discriminant patterns.

Recently ${\ell }_1$-norm-based subspace learning methods haveshown great performance against outliers for tensor data classification (Razzak et al., 0000, Wang et al., 2012, Wang and Wang, 2013). Ke and Kanade presented matrix factorization as an ${\ell }_1$-norm minimization problem that is able to handle missing data straightforwardly. Wang et al. presented robust 2DPCA with non-greedy ${\ell }_1$-norm maximization in which all projection directions are optimized simultaneously (Wang & Gao, 2016). Luo et al. extended it by learning the optimization matrix by maximizing the sum of the projected difference between each pair of instances, rather than the difference between each instance and the mean of the data (Luo et al., 2017). Although, ${\ell }_1$-based methods provided great performance, these methods do not relate to covariance matrix which characterizes the geometric structure of the data, where as F-norm can exploit efficiently the spatial structure that is embedded in the data. Several efforts have been made to utilize F-norm as subspace learning such as 2DPCA (Yang et al., 2004, Yang et al., 2005), 2D-PCA (Tian et al., 2017), F-norm 2DPCA (Li et al., 2017), NM-2DPCA (Chen et al., 2018, Wang et al., 2017), N-2DNPP (Zhang, Li, Zhao, Zhang, & Yan, 2017). However, either these methods still suffer from the effect of outliers or not able to select important features. Furthermore, sensitivity of F-norm is another challenge. Wang et al. presented non-squared F-norm minimization to overcome this challenge (Wang et al., 2017). However, it affects the selection of important features.

To overcome the aforementioned issue of robust feature selection and sensitivity of Frobenius norm, in this paper, we present a novel formulation for PCA that combines the subspace learning and feature selection together in order to exclude the effect of redundant patterns and joint feature selection. We employed Frobenius norm as distance metric learning and seek the projection matrix by joint minimization of regularizer and penalty terms. We relax the orthogonality constraints of transformation matrix and introduce another transformation that helps to jointly select important features and enhances the robustness against outliers. To overcome the sensitivity issue due to squared Frobenius norm, we devised an efficient way to compute F-Norm. As a result, the proposed objective function not only weakens the effect of large distance but also has rotational invariance property. We can describe the theoretical and empirical key contributions of this work as follows:

• We present outliers robust two-dimensional principal component analysis by efficiently integrating the robustness of traditional 2DPCA and the regularization term ${\Vert Q\Vert }_F^2$ that relaxes the orthogonal constraint.

• The regularization term ${\Vert Q\Vert }_F^2$ reduces the constraints and enables the objective function to select features jointly. Furthermore, the regularization parameter ${\Vert Q\Vert }_F^2$ is convex and can be easily optimized.

• To overcome the sensitivity issue of F-Norm against outliers, we efficiently derived the objective function.

• Penalty term penalizes all regression coefficientscorresponding to single feature as a whole to make PCA possible to select features jointly. Hence, ORPCA approximates high-dimensional representation in flexible manner. As such, ORPCA has more freedom to select low-dimensional features efficiently.

• The one major drawback of F-norm is its sensitivity against outliers as outlying measurement arbitrarily skew the solution from desired due to squared objective function. As a result, F-norm is not able to utilize the underlying geometric structure in a real sense. To cope the sensitivity due to squared F-norm, recently, non-square F-norm have been used.

• The latter method is evaluated empirically on four benchmark datasets. Experimental evaluation (discriminant features, computationally and convergence analysis) shows the considerable improvement in most cases, while time complexity remains very attractive.

The rest of the paper is organized as follows. In Section 2, we present basic notations and related work. In Section 3, we present the motivation followed by the proposed objective function and its optimization. In Section 5, we provide detailed experimental evaluations. Finally, conclusion is drawn in Section 6.