Open AccessProceedings Article
A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers
Sahand Negahban,Bin Yu,Martin J. Wainwright,Pradeep Ravikumar +3 more
- 07 Dec 2009
- Vol. 22, pp 1348-1356
TL;DR: A unified framework for establishing consistency and convergence rates for regularized M-estimators under high-dimensional scaling is provided and one main theorem is state and shown how it can be used to re-derive several existing results, and also to obtain several new results.
read more
Abstract: High-dimensional statistical inference deals with models in which the the number of parameters p is comparable to or larger than the sample size n. Since it is usually impossible to obtain consistent procedures unless p/n → 0, a line of recent work has studied models with various types of structure (e.g., sparse vectors; block-structured matrices; low-rank matrices; Markov assumptions). In such settings, a general approach to estimation is to solve a regularized convex program (known as a regularized M-estimator) which combines a loss function (measuring how well the model fits the data) with some regularization function that encourages the assumed structure. The goal of this paper is to provide a unified framework for establishing consistency and convergence rates for such regularized M-estimators under high-dimensional scaling. We state one main theorem and show how it can be used to re-derive several existing results, and also to obtain several new results on consistency and convergence rates. Our analysis also identifies two key properties of loss and regularization functions, referred to as restricted strong convexity and decomposability, that ensure the corresponding regularized M-estimators have fast convergence rates.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Overlapping group lasso for high-dimensional generalized linear models
TL;DR: Under some general conditions, this paper presents the oracle inequalities for the estimation and prediction error of overlapping group Lasso method in the generalized linear model setting and applies these results to the so-called Logistic and Poisson regression models.
5
•Proceedings Article
A nonasymptotic law of iterated logarithm for general M-estimators
Arnak S. Dalalyan,Nicolas Schreuder,Victor-Emmanuel Brunel +2 more
- 03 Jun 2020
TL;DR: We propose the first non-asymptotic deviation bounds for general M-estimators, where "any-time" means that the bound holds with a prescribed probability for every sample size.
•Posted Content
Surrogate Aided Unsupervised Recovery of Sparse Signals in Single Index Models for Binary Outcomes
TL;DR: Under sparsity assumptions, the recovery of regression coefficients is considered for a single index model relating a binary outcome $Y$ to a set of possibly high dimensional covariates $\boldsymbol{X}$, based on a large but 'unlabeled' dataset $\mathcal{U}$, with Y never observed.
5
•Posted Content
Sparse and Robust Linear Regression: An Optimization Algorithm and Its Statistical Properties
Shota Katayama,Hironori Fujisawa +1 more
TL;DR: In this article, a parameter vector for modeling outliers was added to the standard linear regression model and then the sparse estimation problem for both coefficients and outliers is considered, and the algorithm can recover the true support of the coefficients with probability going to one.
5
•Posted Content
High-Dimensional Learning under Approximate Sparsity: A Unifying Framework for Nonsmooth Learning and Regularized Neural Networks
Hongcheng Liu,Yinyu Ye +1 more
TL;DR: In this paper, the authors generalize sparsity and restricted strong convexity (RSC) via the use of the folded concave penalty (FCP), and show that the FCP-based regularization leads to poly-logarithmic sample complexity.
5
References
Regression Shrinkage and Selection via the Lasso
TL;DR: A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
Atomic Decomposition by Basis Pursuit
TL;DR: Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions.
11.3K
Model selection and estimation in regression with grouped variables
Ming Yuan,Yi Lin +1 more
TL;DR: In this paper, instead of selecting factors by stepwise backward elimination, the authors focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection.
Decoding by linear programming
Emmanuel J. Candès,Terence Tao +1 more
TL;DR: F can be recovered exactly by solving a simple convex optimization problem (which one can recast as a linear program) and numerical experiments suggest that this recovery procedure works unreasonably well; f is recovered exactly even in situations where a significant fraction of the output is corrupted.
Exact Matrix Completion via Convex Optimization
TL;DR: It is proved that one can perfectly recover most low-rank matrices from what appears to be an incomplete set of entries, and that objects other than signals and images can be perfectly reconstructed from very limited information.