Machine Learning and Deep Learning-Based Intrusion Detection Systems: A Comprehensive Review of Datasets, Algorithms, Challenges, Explainability, and Future Research Directions

Abstract

Cloud computing, Internet of Things, cyber-physical systems and communications networks are vital technologies with a high growth potential that have made the volume and sophistication of cyberattacks more common and challenging than ever; traditional security mechanisms are no longer adequate for modern threat detection in these rapidly evolving technologies. Machine Learning (ML) and Deep Learning (DL) have proven to be promising solutions to build powerful Intrusion Detection Systems (IDSs) that can detect complex and new attacks. This review will be conducted with a systematic review methodology as guided by PRISMA, and will present the comprehensive analysis of the published ML- and DL-based IDS research from 2020 up to 2026. Over 500 studies were reviewed and 120 high-quality publications were chosen for in-depth review. It reviews popular benchmark datasets, such as NSL-KDD, UNSW-NB15, CICIDS2017, CICDDoS2019, Bot-IoT, IoTID20, and TON_IoT, and analyzes the performances of the popular ML algorithms including SVM, KNN, Random Forest, and XGBoost; and DL architectures including CNN, RNN, LSTM, GRU, Autoencoders, Transformers, and Graph Neural Networks. New areas of research covered include Explainable AI (XAI); Federated Learning; Edge Intelligence; Adversarial Learning and Self-Supervised Learning. The results show that the detection accuracy is generally higher for the DL models, while the interpretability and computational efficiency are higher for the ML models. The following are the top problems found: dataset imbalance, concept drift, adversarial attacks, privacy concerns and real-time deployment constraints. The review then presents the current research opportunities in the field of designing scalable, explainable and trustworthy IDS frameworks in future cybersecurity environments