Open AccessProceedings Article
A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers
Sahand Negahban,Bin Yu,Martin J. Wainwright,Pradeep Ravikumar +3 more
- 07 Dec 2009
- Vol. 22, pp 1348-1356
TL;DR: A unified framework for establishing consistency and convergence rates for regularized M-estimators under high-dimensional scaling is provided and one main theorem is state and shown how it can be used to re-derive several existing results, and also to obtain several new results.
read more
Abstract: High-dimensional statistical inference deals with models in which the the number of parameters p is comparable to or larger than the sample size n. Since it is usually impossible to obtain consistent procedures unless p/n → 0, a line of recent work has studied models with various types of structure (e.g., sparse vectors; block-structured matrices; low-rank matrices; Markov assumptions). In such settings, a general approach to estimation is to solve a regularized convex program (known as a regularized M-estimator) which combines a loss function (measuring how well the model fits the data) with some regularization function that encourages the assumed structure. The goal of this paper is to provide a unified framework for establishing consistency and convergence rates for such regularized M-estimators under high-dimensional scaling. We state one main theorem and show how it can be used to re-derive several existing results, and also to obtain several new results on consistency and convergence rates. Our analysis also identifies two key properties of loss and regularization functions, referred to as restricted strong convexity and decomposability, that ensure the corresponding regularized M-estimators have fast convergence rates.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Dissertation
Robust Semi-Parametric Inference in Semi-Supervised Settings
Abhishek Chakrabortty
- 17 May 2016
TL;DR: This dissertation provides a unified framework for SS M-estimation problems based on general estimating equations, and proposes a family of EASE estimators that are always as efficient as the supervised estimator and more efficient whenever U is actually informative for the parameter of interest.
5
•Posted Content
The Knowledge Gradient Policy Using A Sparse Additive Belief Model
Yan Li,Han Liu,Warren B. Powell +2 more
TL;DR: A knowledge gradient policy for sparse linear models (KGSpLin) with group Lasso penalty is derived, which efficiently learns the correct set of nonzero parameters even when the model is imbedded with hundreds of dummy parameters.
•Posted Content
A Fast and Scalable Joint Estimator for Integrating Additional Knowledge in Learning Multiple Related Sparse Gaussian Graphical Models
TL;DR: In this paper, a hybrid norm is proposed to enforce the superposition of two weighted sparsity constraints, one on the shared interactions and the other on the task-specific structural patterns.
5
•Posted Content
Robust Decoding from 1-Bit Compressive Sampling with Least Squares
TL;DR: This paper introduces a Newton type method, the so-called primal and dual active set (PDAS) algorithm, to solve the nonsmooth optimization problem and proposes a novel regularization parameter selection rule which does not introduce any extra computational overhead.
5
•Posted Content
Regularized Sample Average Approximation for High-Dimensional Stochastic Optimization Under Low-Rankness
TL;DR: The results indicate that high-dimensional low-rank matrix recovery is possible generally beyond a linear model, even if the common assumption of restricted strong convexity is completely absent.
5
References
Regression Shrinkage and Selection via the Lasso
TL;DR: A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
Atomic Decomposition by Basis Pursuit
TL;DR: Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions.
11.3K
Model selection and estimation in regression with grouped variables
Ming Yuan,Yi Lin +1 more
TL;DR: In this paper, instead of selecting factors by stepwise backward elimination, the authors focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection.
Decoding by linear programming
Emmanuel J. Candès,Terence Tao +1 more
TL;DR: F can be recovered exactly by solving a simple convex optimization problem (which one can recast as a linear program) and numerical experiments suggest that this recovery procedure works unreasonably well; f is recovered exactly even in situations where a significant fraction of the output is corrupted.
Exact Matrix Completion via Convex Optimization
TL;DR: It is proved that one can perfectly recover most low-rank matrices from what appears to be an incomplete set of entries, and that objects other than signals and images can be perfectly reconstructed from very limited information.