Anomaly detection using unsupervised machine learning algorithms: A simulation study

Abstract: Abstract There is a growing need for labeled data, yet manual annotation is costly, error-prone, and often infeasible in privacy-sensitive, highly imbalanced domains such as fraud detection. We introduce a fully unsupervised framework that combines unsupervised SHapley Additive exPlanations (SHAP) feature selection with our novel unsupervised labeling method. We apply unsupervised SHAP to the Kaggle Credit Card Fraud Detection and Medicare Part D datasets to produce high-impact feature subsets, and then label the datasets with our unsupervised labeling approach. To effectively evaluate the labels generated by our novel methodology, we apply a baseline unsupervised learner, Isolation Forest (IF), to both the original datasets and their subsets. We calculate Matthew’s Correlation Coefficient (MCC), Jaccard Index (JI), Precision, Recall, and F1-score by comparing our generated labels against the ground truth labels. It is important to note, the ground truth labels were used solely for evaluation. Our empirical results surpass the results obtained with the full feature dataset and baseline. By improving label quality while reducing computational complexity and preserving privacy, our approach offers a practical solution for learning from unlabeled, severely imbalanced data.

TL;DR: This study presents a robust anomaly detection framework, iForest-WGANs, combining Wasserstein GANs with isolation forest algorithms to address missing data in IIoT, achieving 97.2% detection accuracy and 93.9% recall, outperforming conventional benchmarks by 5-12%.

Abstract: The rapid integration of cryptocurrencies into the global financial ecosystem has introduced unprecedented challenges in market surveillance, risk management, and anomaly detection. While conventional statistical models such as ARIMA (Autoregressive Integrated Moving Average) and GARCH (Generalized Autoregressive Conditional Heteroscedasticity) have been widely used for anomaly detection, their reliance on assumptions of normality and stationarity often fails to capture the complexities of high-frequency, non-linear cryptocurrency trading. Furthermore, traditional risk metrics including down-to-up volatility, negative conditional skewness, and relative frequency may overlook short-term anomalies due to data aggregation limitations. In order to address these issues, this paper proposes machine-learning model for detecting anomalies in cryptocurrency markets using Jupyter Notebook. We compare four advanced unsupervised machine learning models, i.e, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Isolation Forest (iForest), One-Class Support Vector Machine (OC-SVM), and Local Outlier Factor (LOF) for anomaly detection by using Monte Carlo simulations. The findings indicate that DBSCAN has the highest precision (79.7%) with the fewest false positives, making it ideal for supervisory monitoring. However, the high false positive rates of OC-SVM and Isolation Forest limit their use. By using data of six well-known cryptocurrencies at three different temporal resolutions (daily, hourly, and 15-minute) the performance of these four unsupervised learning techniques also examined and confirmed that the anomalies identified by DBSCAN are also consistent with the other three methods. Additionally, for robustness of results, we use UpSet Plots to incorporate the shared anomalies and found across the three unsupervised learning methods. Number of anomalies also depends on the volatility and time interval of cryptocurrencies, more volatile / high frequency more anomalies. The study presents sound methodological approach for facilitating financial monitoring and mitigating risks in the cryptocurrencies market, and provides useful information for market players, analysts and policymakers. These results emphasize the importance of choosing algorithms based on specific surveillance targets to promote greater stability in digital asset environments.

TL;DR: The numerical results show that the proposed continuous genetic algorithm is a robust and accurate procedure for solving systems of second-order boundary value problems and the obtained accuracy for the solutions using CGA is much better than the results obtained using some modern methods.

TL;DR: It is argued that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement and it is shown that the optimization problem has a convex equivalent under relatively mild assumptions.

TL;DR: In this article, the authors proposed an attention mechanism-based convolutional neural network-long short-term memory (AMCNN-LSTM) model to accurately detect anomalies.

TL;DR: This survey aims to document the state of anomaly detection in high dimensional big data by representing the unique challenges using a triangular model of vertices: the problem, techniques/algorithms, and tools (big data applications/frameworks).