Improving the Generalization Ability of an Artificial Neural Network in Predicting In-Flight Particle Characteristics of an Atmospheric Plasma Spray Process

Abstract

This paper presents the application of the artificial neural network into an atmospheric plasma spray process for predicting the in-flight particle characteristics, which have significant influence on the in-service coating properties. One of the major problems for such function-approximating neural network is over-fitting, which reduces the generalization capability of a trained network and its ability to work with sufficient accuracy under a new environment. Two methods are used to analyze the improvement in the network’s generalization ability: (i) cross-validation and early stopping, and (ii) Bayesian regularization. Simulations are performed both on the original and expanded database with different training conditions to obtain the variations in performance of the trained networks under various environments. The study further illustrates the design and optimization procedures and analyzes the predicted values, with respect to the experimental ones, to evaluate the performance and generalization ability of the network. The simulation results show that the performance of the trained networks with regularization is improved over that with cross-validation and early stopping and, furthermore, the generalization capability of the networks is improved; thus preventing any phenomenon associated with over-fitting.

Appendices

Appendix A1: Database (DS_O)

Table 5

Appendix A2: Generalization Errors Generated by Networks Trained by Levenberg-Marquardt Algorithm with the Datasets DS_OTR and DS_ETR

Table 6

Appendix A3: Levenberg-Marquardt Algorithm

Levenberg-Marquardt algorithm is an approximation to the Newton’s method and is designed to reach the second order training speed without computing the Hessian matrix. The approximation of the Hessian matrix and the error gradient is computed as per Eq 5 and 6.

$$ H = J^{T} J $$

(10)

$$ g = J^{\text{T}} e $$

(11)

J represents the Jacobian matrix formed with the first derivatives of the network errors e, on the training set with respect to the network’s weights and biases and can be calculated using standard back propagation technique (Ref 21). The Levenberg-Marquardt algorithm uses the approximation in calculation of the Hessian matrix to update and tune the parameters. If z _k represents the old parameter value, then the new parameter value after calculation of the network errors, is given by Eq 7.

$$ z_{k + 1} = z_{k} - [J^{\text{T}} J + \mu I]^{ - 1} J^{\text{T}} e $$

(12)

The parameter μ is set to a specific value at the start of the training. After each epoch, the performance function is computed. If the performance function is found out to be less than the previous epoch, the value of μ is decreased by a specific value. However, if the performance function increases, the value of μ is also increased by a specific value. Having the value of μ equals zero, turns Eq 7 into a Newton’s method. The aim is to shift toward Newton’s method rapidly since it is faster and more accurate near minimum error. A maximum value of μ is set before the training. If μ reaches its maximum value, the training stops and it indicates that the network has failed to converge. The training is also stopped when the error gradient (Eq 6) falls below a specific set value or when the goal set for the performance function is met.

The training set is used during the network learning stage to calculate the error gradient (Eq 6) and update the network’s weights and biases accordingly. The validation and the test sets are never used to update network’s weights and biases. The network’s error on the validation set is calculated and monitored during the training process. As the training starts, the validation set error decreases along with the training set error. However, as the training progresses, the error on the validation set starts to rise and if the error increases for a specific number of epochs (iterations), it indicates that the network has started to over fit. The training is thus immediately stopped and the values of the weights and biases, during the minimum validation error, are returned and saved. The test set is only used to conduct the performance evaluation of the trained network. In order to achieve statistically significant results, several independent data splits must be performed followed by lengthy trainings.