Journal Article10.1109/idap64064.2024.10710799
Anomaly Detection on Servers Using Log Analysis
Mert İnan Saygılı,Saltuk Buğra Özelgül,İlkay Samet Öztürk,Kevser Özdem,Ahmet Orkun Gedik,M. Ali Akçayol +5 more
- 21 Sep 2024
pp 1-5
TL;DR: This study develops a deep learning model using a Convolutional Neural Network (CNN) to detect anomalies in server log data, achieving up to 99% accuracy rates and improving debugging and operating efficiency in Hadoop Distributed File System (HDFS) environments.
read more
Abstract: Increasing data volume and complexity make log analysis mandatory for security and performance management in server systems. In this new era, where traditional manual methods are insufficient, the automatic log analysis potential of artificial intelligence and deep learning techniques comes to the fore. In this study, a deep learning model is developed to detect anomalies by analyzing log data collected from servers and devices. This log anomaly detection model, developed using a Convolutional Neural Network (CNN), uses structured log data processed with the Drain log parsing algorithm and effectively classifies anomalies by extracting features from this data. In the experimental studies conducted on Hadoop Distributed File System (HDFS) log data, it is observed that the model reaches up to $99 \%$ accuracy rates and improves both debugging processes and operating efficiency.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
References
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Min Du,Feifei Li,Guineng Zheng,Vivek Srikumar +3 more
- 30 Oct 2017
TL;DR: DeepLog, a deep neural network model utilizing Long Short-Term Memory (LSTM), is proposed, to model a system log as a natural language sequence, which allows DeepLog to automatically learn log patterns from normal execution, and detect anomalies when log patterns deviate from the model trained from log data under normal execution.
1.4K
Mining anomalies using traffic feature distributions
Anukool Lakhina,Mark Crovella,Christophe Diot +2 more
- 22 Aug 2005
TL;DR: It is argued that the distributions of packet features observed in flow traces reveals both the presence and the structure of a wide range of anomalies, and that using feature distributions, anomalies naturally fall into distinct and meaningful clusters that can be used to automatically classify anomalies and to uncover new anomaly types.
Drain: An Online Log Parsing Approach with Fixed Depth Tree
Pinjia He,Jieming Zhu,Zibin Zheng,Michael R. Lyu +3 more
- 25 Jun 2017
TL;DR: This work proposes an online log parsing method, namely Drain, that can parse logs in a streaming and timely manner, and uses a fixed depth parse tree, which encodes specially designed rules for parsing.
714
Anomaly detection in IP networks
Marina Thottan,Chuanyi Ji +1 more
TL;DR: A statistical signal processing technique based on abrupt change detection is described that is effective at detecting several network anomalies and has great potential to enhance the field, and thereby improve the reliability of IP networks.
Tools and benchmarks for automated log parsing
Jieming Zhu,Shilin He,Jinyang Liu,Pinjia He,Qi Xie,Zibin Zheng,Michael R. Lyu +6 more
- 27 May 2019
TL;DR: This paper presents a comprehensive evaluation study on automated log parsing, evaluating 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software and reports the results in terms of accuracy, robustness, and efficiency.
484