An iterative pruning algorithm for feedforward neural networks

Question

1. What are the contributions mentioned in the paper "An iterative pruning algorithm for feedforward neural networks" ?

2. What is the approach to network pruning?

3. What is the way to evaluate the performance of the proposed method?

4. What is the meaning of the pruning algorithm?

Accepted Answer

One popular approach tackling this problem is commonly known as pruning and consists of training a larger than necessary network and then removing unnecessary weights/nodes.. In this paper, a new pruning method is developed, based on the idea of iteratively eliminating units and adjusting the remaining weights in such a way that the network performance does not worsen over the entire training set.. The algorithm also provides a simple criterion for choosing the units to be removed, which has proved to work well in practice.

Accepted Answer

Their approach to network pruning consists first of removing unit and then appropriately adjusting the weights incoming into ’s projective field so as to preserve the overall network input/output behavior of the training set.

Accepted Answer

To assess the performance of the proposed method, the well-known parity and symmetry tasks [2] were chosen, as the near-optimal number of hidden units required to achieve their solution is known.

Accepted Answer

Since the pruning algorithm presented in this paper can be applied to arbitrary feedforward networks, not necessarily layered or fully connected, some definitions and notations must be introduced.

Accepted Answer

To solve the least-squares problem (14), standard QR factorization methods with column pivoting or the SVD technique can be employed [41].

Accepted Answer

This paper describes a method for reducing the size of trained feedforward neural networks, in which the key ideaconsists of iteratively removing hidden units and then adjusting the remaining weights in such a way as to preserve the overall network behavior.

Accepted Answer

to evaluate the behavior of their pruning procedure under “stressed” conditions, each trained network was pruned (regardless of its performance) until the hidden layer contained exactly one unit.

Accepted Answer

It is clear that the major drawback of S&D’s approach is its simplicity: in real-world applications, in fact, hidden units are unlikely to be exactly correlated and the algorithm may thus fail to detect redundant elements.

Accepted Answer

In order to maintain the original network behavior, the authors attempt to adjust the remaining weights incoming into node i.e., the ’s for all so that its new net input remains as close as possible to the old one, for all the training patterns (see Fig. 2).

Accepted Answer

it is well known that good generalization results are typically achieved when performance over the learning set is nonoptimal [53]–[55].

Accepted Answer

In addition, for initially small networks, convergence is not always guaranteed; in fact, seven out of ten parity trials did not converge within 3000 epochs.

Accepted Answer

the authors point out that the generalization results of the networks reduced by their pruning algorithm are not only superior to those found by Holt [51] under experimental conditions similar to ours (i.e., same classification problem, equal training and test set sizes, network architecture, and learning stopping criterion), but are comparable with the best results he obtained using an alternative cost function aiming to improve the generalization performance.

Accepted Answer

To evaluate the behavior of both methods, three different measures were adopted: 1) the number of hidden nodes in the reduced networks; 2) the recognition rate, measured as the proportion of examples for which all network output values differed from the corresponding target by less than 0.5; and 3) the usual mean-squared error (MSE).

Accepted Answer

The approach to network pruning pursued in this paper offers a general and systematic method for reducing the size of a trained network which contrasts sharply with the heuristic, ad hoc procedure developed by Sietsma and Dow: ours, in fact, is not based on any simplifying assumption, does not depend on any working parameter, and does not demand additional retraining sessions.

Accepted Answer

the number of cycles performed by the CGPCNE procedure, like any CGbased least-squares method [42], [43], turns out to be very low and typically far less than either or so that the overall computational complexity of each pruning step can be approximately

Accepted Answer

The original networks performed better than the pruned ones over the training set, due to the particular rule employed to stop pruning, which does not take into account the performance of the reduced networks over the training data as pruning proceeds.

Accepted Answer

to compare the time required to train and prune a large solution network with that for directly training a small network, ten independent 4-5-1 and 4-4-1 networks weretrained for parity and symmetry respectively; the size of these networks corresponds to the “median” sizes found by their procedure.

Accepted Answer

This leads to formulating the pruning problem in terms of defining a system of linear equations that the authors solve with a very efficient conjugate gradient least-squares procedure.

An iterative pruning algorithm for feedforward neural networks

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What are the contributions mentioned in the paper "An iterative pruning algorithm for feedforward neural networks" ?

2. What is the approach to network pruning?

3. What is the way to evaluate the performance of the proposed method?

4. What is the meaning of the pruning algorithm?

5. What is the common method used to solve the least-squares problem?

6. What is the main idea of the paper?

7. How many hidden units were removed from the network?

8. What is the main drawback of S&D’s approach?

9. What is the simplest way to maintain the original network behavior?

10. What is the generalization result for the pattern recognition problem?

11. How many trials did the pruning algorithm fail to converge within 3000 epochs?

12. What is the generalization algorithm for pruning?

13. What was the way to evaluate the behavior of the two methods?

14. What is the main drawback of the approach to network pruning?

15. How many cycles does the algorithm take to prune a network?

16. Why did the original networks perform better than the pruned ones?

17. How many trials were required to train a large solution network?

18. How does the proposed pruning procedure work?

Figures

Citations

Statistical pattern recognition: a review

Supervised Machine Learning: A Review of Classification Techniques

Neural networks for the prediction and forecasting of water resources variables: a review of modelling issues and applications

Machine learning: a review of classification and combining techniques

Structured Pruning of Deep Convolutional Neural Networks

References

Learning representations by back-propagating errors

Classification and regression trees

Classification and Regression Trees.

Learning internal representations by error propagation

Learning internal representations by error propagation

Related Papers (5)

Deep Residual Learning for Image Recognition

Learning internal representations by error propagation

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Gradient-based learning applied to document recognition

Neural Networks: A Comprehensive Foundation