Proceedings Article10.1145/3379597.3387448
AIMMX: Artificial Intelligence Model Metadata Extractor
Jason Tsay,Alan Braz,Martin Hirzel,Avraham Shinnar,Todd W. Mummert +4 more
- 29 Jun 2020
- pp 81-92
21
TL;DR: An exploratory analysis for data and method reproducibility over the models in the evaluation dataset and a catalog tool for discovering and managing models that enables simplified AI Model Metadata eXtraction from software repositories are presented.
read more
Abstract: Despite all of the power that machine learning and artificial intelligence (AI) models bring to applications, much of AI development is currently a fairly ad hoc process. Software engineering and AI development share many of the same languages and tools, but AI development as an engineering practice is still in early stages. Mining software repositories of AI models enables insight into the current state of AI development. However, much of the relevant metadata around models are not easily extractable directly from repositories and require deduction or domain knowledge. This paper presents a library called AIMMX that enables simplified AI Model Metadata eXtraction from software repositories. The extractors have five modules for extracting AI model-specific metadata: model name, associated datasets, references, AI frameworks used, and model domain. We evaluated AIMMX against 7,998 open-source models from three sources: model zoos, arXiv AI papers, and state-of-the-art AI papers. Our platform extracted metadata with 87% precision and 83% recall. As preliminary examples of how AI model metadata extraction enables studies and tools to advance engineering support for AI development, this paper presents an exploratory analysis for data and method reproducibility over the models in the evaluation dataset and a catalog tool for discovering and managing models. Our analysis suggests that while data reproducibility may be relatively poor with 42% of models in our sample citing their datasets, method reproducibility is more common at 72% of models in our sample, particularly state-of-the-art models. Our collected models are searchable in a catalog that uses existing metadata to enable advanced discovery features for efficiently finding models.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
A Survey of Privacy Risks and Mitigation Strategies in the Artificial Intelligence Life Cycle
01 Jan 2023
TL;DR: In this paper , the authors examine privacy risks in different phases of the AI life cycle and review the existing privacy-enhancing solutions, including technologies, requirements, and process solutions to countermeasure these risks.
How do Hugging Face Models Document Datasets, Bias, and Licenses? An Empirical Study
Federica Pepe,Vittoria Nardone,Antonio Mastropaolo,Gabriele Bavota,Gerardo Canfora,Massimiliano Di Penta +5 more
- 15 Apr 2024
TL;DR: Pre-trained transformer models hosted by Hugging Face lack transparency in terms of datasets, bias, and licenses. There is a need for further research and potential legislation to improve the transparency of ML models.
14
Complex Python Features in the Wild
Yi Yang,Ana Milanova,Martin Hirzel +2 more
- 01 May 2022
TL;DR: The findings show that usage of dynamic features that pose a threat to static analysis is infrequent, and a list of Python features that any “minimal syntax” ought to handle in order to capture developers' use of the Python language.
13
Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability
Diego Montes,Pongpatapee Peerapatanapokin,Jeff Schultz,Chengjun Guo,Wenxin Jiang,James C. Davis +5 more
- 07 Nov 2022
TL;DR: In this paper , the reliability of pre-trained deep neural networks (DNNs) from model zoos is investigated. But, the authors focus on the accuracy, latency, and architecture of 36 DNNs across four model zooms, and find differences of 1.23%-2.62% in accuracy and 9%-131% in latency.
References
•Posted Content
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
TL;DR: A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
81.7K
•Proceedings Article
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Christian Szegedy,Sergey Ioffe,Vincent Vanhoucke,Alexander A. Alemi +3 more
- 23 Feb 2016
TL;DR: In this paper, the authors show that training with residual connections accelerates the training of Inception networks significantly, and they also present several new streamlined architectures for both residual and non-residual Inception Networks.
•Proceedings Article
Hidden technical debt in Machine learning systems
D. Sculley,Gary Holt,Daniel Golovin,Eugene Davydov,Todd Phillips,Dietmar Ebner,Vinay Chaudhary,Michael Young,Jean-Francois Crespo,Dan Dennison +9 more
- 07 Dec 2015
TL;DR: It is found it is common to incur massive ongoing maintenance costs in real-world ML systems, and several ML-specific risk factors to account for in system design are explored.
One billion word benchmark for measuring progress in statistical language modeling.
Ciprian Chelba,Tomas Mikolov,Mike Schuster,Qi Ge,Thorsten Brants,Phillipp Koehn,Tony Robinson +6 more
- 14 Sep 2014
TL;DR: A new benchmark corpus to be used for measuring progress in statistical language modeling, with almost one billion words of training data, is proposed, which is useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques.
1K