Efficient Decentralized Deep Learning by Dynamic Model Averaging
Michael Kamp,Linara Adilova,Joachim Sicking,Fabian Hüger,Peter Schlicht,Tim Wirtz,Stefan Wrobel +6 more
- 10 Sep 2018
- pp 393-409
117
TL;DR: Kamp et al. as mentioned in this paper proposed an efficient protocol for decentralized training of deep neural networks from distributed data sources, which allows to handle different phases of model training equally well and to quickly adapt to concept drifts.
read more
Abstract: We propose an efficient protocol for decentralized training of deep neural networks from distributed data sources. The proposed protocol allows to handle different phases of model training equally well and to quickly adapt to concept drifts. This leads to a reduction of communication by an order of magnitude compared to periodically communicating state-of-the-art approaches. Moreover, we derive a communication bound that scales well with the hardness of the serialized learning problem. The reduction in communication comes at almost no cost, as the predictive performance remains virtually unchanged. Indeed, the proposed protocol retains loss bounds of periodically averaging schemes. An extensive empirical evaluation validates major improvement of the trade-off between model performance and communication which could be beneficial for numerous decentralized learning applications, such as autonomous driving, or voice recognition and image classification on mobile phones. Code related to this paper is available at: https://bitbucket.org/Michael_Kamp/decentralized-machine-learning.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Posted Content
LEAF: A Benchmark for Federated Settings
Sebastian Caldas,Peter Wu,Tian Li,Jakub Konečný,H. Brendan McMahan,Virginia Smith,Ameet Talwalkar +6 more
TL;DR: LEAF is proposed, a modular benchmarking framework for learning in federated settings that includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.
1K
Federated Learning for Healthcare Informatics
Jie Xu,Benjamin S. Glicksberg,Chang Su,Peter B. Walker,Jian-Guo Bian,Fei Wang +5 more
- 01 Mar 2021
TL;DR: In this article, the authors provide a review of federated learning in the biomedical space, and summarize the general solutions to the statistical challenges, system challenges, and privacy issues in federated Learning, and point out the implications and potentials in healthcare.
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning
Hao Yu,Sen Yang,Shenghuo Zhu +2 more
- 17 Jul 2019
TL;DR: A thorough and rigorous theoretical study on why model averaging can work as well as parallel mini-batch SGD with significantly less communication overhead.
•Posted Content
Federated Learning for Healthcare Informatics
TL;DR: The goal of this survey is to provide a review for federated learning technologies, particularly within the biomedical space, and summarize the general solutions to the statistical challenges, system challenges, and privacy issues in federation, and point out the implications and potentials in healthcare.
683
Broadband Analog Aggregation for Low-Latency Federated Edge Learning
TL;DR: In this paper, a low-latency multi-access scheme for edge learning is proposed, where the updates simultaneously transmitted by devices over broadband channels should be analog aggregated "over-the-air" by exploiting the waveform-superposition property of a multiaccess channel.
633
References
Understanding deep learning (still) requires rethinking generalization
TL;DR: These experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data, and confirm that simple depth two neural networks already have perfect finite sample expressivity.
•Posted Content
End to End Learning for Self-Driving Cars
Mariusz Bojarski,Davide Del Testa,Daniel Dworakowski,Bernhard Firner,Beat Flepp,Prasoon Goyal,Lawrence D. Jackel,Mathew Monfort,Urs A. Muller,Jiakai Zhang,Xin Zhang,Jake Zhao,Karol Zieba +12 more
TL;DR: A convolutional neural network is trained to map raw pixels from a single front-facing camera directly to steering commands and it is argued that this will eventually lead to better performance and smaller systems.
•Proceedings Article
Large Scale Distributed Deep Networks
Jeffrey Dean,Greg S. Corrado,Rajat Monga,Kai Chen,Matthieu Devin,Mark Z. Mao,Marc'Aurelio Ranzato,Andrew W. Senior,Paul A. Tucker,Ke Yang,Quoc V. Le,Andrew Y. Ng +11 more
- 03 Dec 2012
TL;DR: This paper considers the problem of training a deep network with billions of parameters using tens of thousands of CPU cores and develops two algorithms for large-scale distributed training, Downpour SGD and Sandblaster L-BFGS, which increase the scale and speed of deep network training.
•Proceedings Article
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan,Eider Moore,Daniel Ramage,Seth Hampson,Blaise Aguera y Arcas +4 more
- 10 Apr 2017
TL;DR: In this paper, the authors presented a decentralized approach for federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets.
ALVINN: An Autonomous Land Vehicle in a Neural Network
Dean A. Pomerleau
- 01 Jan 1988
TL;DR: ALVINN (Autonomous Land Vehicle In a Neural Network) is a 3-layer back-propagation network designed for the task of road following that can effectively follow real roads under certain field conditions.