Machine Learning Techniques to Detect Known and Novel Cyber-attacks

Intrusion detection systems are considered well-known tools for monitoring and detecting malicious traffic in communication networks. However, traditional intrusion detection systems rely on known signatures and lack the ability to detect novel attacks. Therefore, machine learning techniques are introduced to complement intrusion detection and to dynamically identify the relevant data of interest and intelligently find out the security threats. However, in order to train algorithms in machine learning based intrusion detection systems, obtaining reliable datasets with appropriate characteristics is a major challenge. Due to the lack of labelled datasets, machine learning based intrusion detection systems suffer from overfitting problem which makes them inefficient for real time intrusion detection. Furthermore, in real-life scenarios, considerable amount of incoming data does not belong to any known category; and for unknown traffic, dividing data into the classes without having information on the nature of the traffic is challenging. In addition, annotating a large dataset is very costly and hence in practice we can label only a few examples manually. On the other hand, the 5G+ and 6G networks are expected to deliver massive connectivity to numerous IoT/IoE devices, where a huge amount of data needs to be analyzed by artificial intelligence enabled mechanisms. Consequently, a mature and scalable architecture must be considered as a mandatory objective in machine learning based intrusion detection systems.This thesis explores machine learning techniques to handle mentioned issues in the cyber-security domain. The thesis proposes an intelligent, modular, robust and scalable security solution to dynamically detect known and unknown cyber-attacks targeting mobile networks. This project takes the intrusion detection to the next level with a hybrid machine learning based mechanism namely Hybrid Anomaly Detection Model that employs a protocol analyzer and various supervised and unsupervised techniques to filter network traffic and identify malicious activities in high load communication networks. The protocol analyzer classifies and filters vulnerable protocols to avoid unnecessary computation load, the classifiers detect known cyber-attacks, while clustering algorithms use these attributes and features to detect novel attacks.