A new nonsmooth optimization algorithm for minimum sum-of-squares clustering problems

Abstract

The minimum sum-of-squares clustering problem is formulated as a problem of nonsmooth, nonconvex optimization, and an algorithm for solving the former problem based on nonsmooth optimization techniques is developed. The issue of applying this algorithm to large data sets is discussed. Results of numerical experiments have been presented which demonstrate the effectiveness of the proposed algorithm.

Introduction

Clustering is the unsupervised classification of the patterns. Cluster analysis deals with the problems of organization of a collection of patterns into clusters based on similarity. It has found many applications, including information retrieval, document extraction, image segmentation, etc.

In cluster analysis we assume that we have been given a set X of a finite number of points of d-dimensional space ${\mathbb{R}}^d$, that is

$$X=\{x^1\text{,}\dots \text{,}{x}^n\}\text{,}\mathrm{where}{x}^i\in {\mathbb{R}}^d\text{,}i=1\text{,}\dots \text{,}n\text{.}$$

The subject of cluster analysis is the partition of the set X into a given number q of overlapping or disjoint subsets C_i, i = 1,…, q, with respect to predefined criteria such that

$$X=\bigcup _{i=1}^q{C}_i\text{.}$$

The sets C_i, i = 1, … , q, are called clusters. The clustering problem is said to be hard clustering if every data point belongs to one and only one cluster. Unlike hard clustering in the fuzzy clustering problem the clusters are allowed to overlap and instances have degrees of appearance in each cluster. In this paper we will exclusively consider the hard unconstrained clustering problem, that is we additionally assume that

$$C_i\cap C_k=\varnothing \text{,}\forall i\text{,}k=1\text{,}\dots \text{,}q\text{,}i\ne k\text{,}$$

and no constraints are imposed on the clusters C_i, i = 1, … , q. Thus every point x∈X is contained in exactly one and only one set C_i.

Each cluster C_i can be identified by its center (or centroid). Then the clustering problem can be reduced to the following optimization problem (see [7], [8], [38]):

$$\begin{array}{cc}\hfill & \mathrm{minimize}\varphi (C\text{,}a)=\frac{1}{n}\sum _{i=1}^q\sum _{x\in C_i}\parallel a^i-x{\parallel }^2\hfill \\ \hfill & \mathrm{subject}\mathrm{to}C\in \overline{C}\text{,}a=(a^1\text{,}\dots \text{,}{a}^q)\in {\mathbb{R}}^{d\times q}\text{,}\hfill \end{array}$$

where ∥ · ∥ denotes the Euclidean norm, C = {C₁, … , C_q} is a set of clusters, $\overline{C}$ is a set of all possible q-partitions of the set X, aⁱ is the center of the cluster C_i, i = 1, … , q,

$$a^i=\frac{1}{|C_i|}\sum _{x\in C_i}x\text{,}$$

and ∣C_i∣ is a cardinality of the set C_i, i = 1, … , q. The problem (1) is also known as the minimum sum-of-squares clustering. The combinatorial formulation (1) of the minimum sum-of-squares clustering is not suitable for direct application of mathematical programming techniques. The problem (1) can be rewritten as the following mathematical programming problem:

$$\begin{array}{cc}\hfill & \mathrm{minimize}\psi (a\text{,}w)=\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{w}_{\mathit{ij}}\parallel a^j-x^i{\parallel }^2\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\sum _{j=1}^q{w}_{\mathit{ij}}=1\text{,}i=1\text{,}\dots \text{,}n\text{,}\hfill \end{array}$$

and

$$w_{\mathit{ij}}\in \{0\text{,}1\}\text{,}i=1\text{,}\dots \text{,}n\text{,}j=1\text{,}\dots \text{,}q\text{.}$$

Here

$$a^j=\frac{{\sum }_{i=1}^n{w}_{\mathit{ij}}{x}^j}{{\sum }_{i=1}^n{w}_{\mathit{ij}}}\text{,}j=1\text{,}\dots \text{,}q\text{,}$$

and w_ij is the association weight of pattern xⁱ with cluster j (to be found), given by

$$w_{\mathit{ij}}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{pattern}i\mathrm{is}\mathrm{allocated}\mathrm{to}\mathrm{cluster}j\forall i=1\text{,}\dots \text{,}n\text{,}j=1\text{,}\dots \text{,}q\text{,}\hfill \\ 0\hfill & \mathrm{otherwise}\text{,}\hfill \end{array}\right.$$

w is an n × q matrix.

There exist different approaches to clustering including agglomerative and divisive hierarchical clustering algorithms as well as algorithms based on mathematical programming techniques. Descriptions of many of these algorithms can be found, for example, in [14], [23], [38]. An excellent up-to-date survey of existing approaches is provided in [24] and a comprehensive list of literature on clustering algorithms is available in this paper.

Problem (2) is a global optimization problem. Therefore different algorithms of mathematical programming can be applied to solve this problem. Some review of these algorithms can be found in [18] with dynamic programming, branch and bound, cutting planes, k-means algorithms being among them. Dynamic programming approach can be effectively applied to the clustering problem when the number of instances n⩽20, which means that this method is not effective to solve real-world problems (see [25]). However, when q = 1 the clustering problem can be solved exactly by dynamic programming, in polynomial time [38].

Branch and bound algorithms are effective when the database contain only hundreds of records and the number of clusters is not large (less than 5) (see [13], [17], [18], [27]). For these methods the solution of clustering problems with n ⩾ 1000 and q ⩾ 10 is out of reach. Different heuristics can be used for solving large clustering problems and k-means is one such algorithm. Different versions of this algorithm have been studied by many authors (see [38]). This is a very fast algorithm and it is suitable for solving clustering problems in large data sets. k-means gives good results when there are few clusters but deteriorates when there are many [18]. This algorithm achieves a local minimum of problem (1) (see [36]), however, results of numerical experiments presented, for example, in [21] show that the best clustering found with k-means may be more than 50% worse than the best known one.

Much better results have been obtained with metaheuristics, such as simulated annealing, tabu search and genetic algorithms [34]. The simulated annealing approaches to clustering have been studied, for example, in [9], [37], [39]. Application of tabu search methods for solving clustering problem is studied in [1]. Genetic algorithms for clustering have been described in [34]. The results of numerical experiments, presented in paper [2] show that even for small problems of cluster analysis when the number of entities n⩽100 and the number of clusters q⩽5 these algorithms take 500–700 (sometimes several thousands) times more CPU time than the k-means algorithms. For relatively large databases one can expect that this difference will increase. This makes metaheuristic algorithms of global optimization ineffective for solving many clustering problems. However, these algorithms can be applied to large clustering problems if combined with decomposition (see [20]).

An approach to cluster analysis problems based on bilinear programming techniques has been described in [29]. The paper [5] describes the global optimization approach to clustering and demonstrates how the supervised data classification problem can be solved via clustering. The objective function in this problem is both nonsmooth and nonconvex and this function has a large number of local minimizers. Problems of this type are quite challenging for general-purpose global optimization techniques. Due to the large number of variables and the complexity of the objective function these techniques, as a rule, fail to solve such problems.

In [15] an interior point method for minimum sum-of-squares clustering problem is developed. The papers [20], [30] develops variable neighborhood search algorithm and the paper [19] presents j-means algorithm which extends k-means by adding a jump move. The global k-means heuristic, which is an incremental approach to minimum sum-of-squares clustering problem, is developed in [28]. The incremental approach is also studied in the paper [21]. Results of numerical experiments presented show the high effectiveness of these algorithms for many clustering problems.

As mentioned above, the problem (2) is the global optimization problem and the objective function in this problem has many local minima. However, global optimization techniques are highly time-consuming for solving many clustering problems. It is very important, therefore, to develop clustering algorithms based on optimization techniques that compute “deep” local minimizers of the objective function. The clustering algorithm proposed and studied in this paper is of this type and is based on nonsmooth optimization techniques. The algorithm provides the capability of calculating clusters step-by-step, gradually increasing the number of data clusters until termination conditions are met, that is it allows one to calculate as many cluster as a data set contains with respect to some tolerance.

The paper is organized as follows: the nonsmooth optimization approach to clustering is presented in Section 2. Section 3 describes an algorithm for solving clustering problems. An algorithm for solving optimization problems is discussed in Section 4. The issues of the complexity reduction for clustering in large data set are discussed in Section 5; while Section 6 presents discussion of the results of the numerical experiments. Section 7 concludes the paper.