A kernel path algorithm for support vector machines
Gang Wang,Dit-Yan Yeung,Frederick H. Lochovsky +2 more
- 20 Jun 2007
- pp 951-958
TL;DR: This paper learns the hyperparameter of the kernel function for a support vector machine (SVM) without having to train the model multiple times, and finds that the solutions of the neighborhood hyperparameters can be calculated exactly.
read more
Abstract: The choice of the kernel function which determines the mapping between the input space and the feature space is of crucial importance to kernel methods. The past few years have seen many efforts in learning either the kernel function or the kernel matrix. In this paper, we address this model selection issue by learning the hyperparameter of the kernel function for a support vector machine (SVM). We trace the solution path with respect to the kernel hyperparameter without having to train the model multiple times. Given a kernel hyperparameter value and the optimal solution obtained for that value, we find that the solutions of the neighborhood hyperparameters can be calculated exactly. However, the solution path does not exhibit piecewise linearity and extends nonlinearly. As a result, the breakpoints cannot be computed in advance. We propose a method to approximate the breakpoints. Our method is both efficient and general in the sense that it can be applied to many kernel functions in common use.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Figure 3. Change in (a) the generalization accuracy and (b) the number of SVs and (c) the number of events occurred along the kernel path with different λ values for the two-moons dataset. The horizontal axis is in logarithm scale. The generalization accuracy first increases and then decreases and the number of SVs first decreases and then increases as γ decreases from 10 to 0.2. 
Figure 2. SVM classification results of the two-moons dataset for different kernel hyperparameter values (γ = 5, 2, 0.5). In each sub-figure, the middle line (green) shows the decision boundary and the other two lines specify the margins with the points it contains indicated by blue circles. λ is set to 1. 
Figure 1. Number of iterations logθ π + log2(log1− θ) vs. decay rate θ. Three different π values (0.85, 0.9, 0.95) are considered and the error tolerance is set to 10−6. 
Table 1. Kernel path algorithm. 
Figure 4. SVM classification results of the two-Gaussians dataset for different kernel hyperparameter values (γ = 5, 2, 0.5). λ is set to 1. 
Figure 5. Change in (a) the generalization accuracy and (b) the number of SVs and (c) the number of events occurred along the kernel path with different λ values for the two-Gaussians dataset. The horizontal axis is in logarithm scale. The generalization accuracy decreases and the number of SVs increases as γ decreases from 10 to 0.2.
Citations
Incremental Support Vector Learning for Ordinal Regression
TL;DR: Numerical experiments on the several benchmark and real-world data sets show that the incremental algorithm can converge to the optimal solution in a finite number of steps, and is faster than the existing batch and incremental SVOR algorithms.
750
Sparse convex optimization methods for machine learning
Martin Jaggi
- 01 Jan 2011
TL;DR: A convergence proof guaranteeing e-small error is given after O( 1e ) iterations, and the sparsity of approximate solutions for any `1-regularized convex optimization problem (and for optimization over the simplex), expressed as a function of the approximation quality.
Analysis of the Distance Between Two Classes for Tuning SVM Hyperparameters
TL;DR: This paper proposes a novel method for tuning the hyperparameters by maximizing the distance between two classes (DBTC) in the feature space by developing a gradient-based algorithm to search the optimal kernel parameter.
63
An Improved Algorithm for the Solution of the Regularization Path of Support Vector Machine
TL;DR: An improved algorithm for the numerical solution to the support vector machine (SVM) classification problem for all values of the regularization parameter C that follows the main idea of tracking the optimality conditions of the SVM solution for ascending value of C.
62
Approximating parameterized convex optimization problems
Joachim Giesen,Martin Jaggi,Sören Laue +2 more
- 06 Sep 2010
TL;DR: A simple and efficient scheme for maintaining an e-approximate solution (and a corresponding e-coreset) along the entire parameter path for parameterized convex optimization problems over the unit simplex, that depend on one parameter.
References
LIBSVM: A library for support vector machines
Chih-Chung Chang,Chih-Jen Lin +1 more
TL;DR: Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.
Least angle regression
Bradley Efron,Trevor Hastie,Iain M. Johnstone,Robert Tibshirani,Hemant Ishwaran,Keith Knight,Jean-Michel Loubes,Jean-Michel Loubes,Pascal Massart,Pascal Massart,David Madigan,David Madigan,Greg Ridgeway,Greg Ridgeway,Saharon Rosset,Saharon Rosset,Ji Zhu,Robert A. Stine,Berwin A. Turlach,Sanford Weisberg +19 more
TL;DR: A publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates is described.
Least Angle Regression
TL;DR: Least Angle Regression (LARS) as discussed by the authors is a new model selection algorithm, which is a useful and less greedy version of traditional forward selection methods such as All Subsets, Forward Selection and Backward Elimination.
An introduction to kernel-based learning algorithms
TL;DR: This paper provides an introduction to support vector machines, kernel Fisher discriminant analysis, and kernel principal component analysis, as examples for successful kernel-based learning methods.
Learning the Kernel Matrix with Semidefinite Programming
Gert R. G. Lanckriet,Nello Cristianini,Peter L. Bartlett,Laurent El Ghaoui,Michael I. Jordan +4 more
TL;DR: This paper shows how the kernel matrix can be learned from data via semidefinite programming (SDP) techniques and leads directly to a convex method for learning the 2-norm soft margin parameter in support vector machines, solving an important open problem.
Related Papers (5)
Vladimir Vapnik
- 01 Jan 1998
Bradley Efron,Trevor Hastie,Iain M. Johnstone,Robert Tibshirani,Hemant Ishwaran,Keith Knight,Jean-Michel Loubes,Jean-Michel Loubes,Pascal Massart,Pascal Massart,David Madigan,David Madigan,Greg Ridgeway,Greg Ridgeway,Saharon Rosset,Saharon Rosset,Ji Zhu,Robert A. Stine,Berwin A. Turlach,Sanford Weisberg +19 more
Vladimir Vapnik
- 01 Jan 1995