T. V. JAMGHARYAN, Lieutenant Colonel (Res.), PhD in Engineering, Senior Technical Consultant at NSM Security (R&D Department), Lecturer, National Polytechnic University of Armenia;

T. N. SHAHNAZARYAN, Major General, Head, Main Operative Department ­– Deputy Chief, General Staff of the RA Armed Forces

Introduction

The Machine Learning (ML) technologies have rapidly penetrated   all areas of human activity, including healthcare, transport, economics, industry, and, most importantly, the defense sector. Active use of ML is observed in military technologies: in the control of unmanned aerial vehicles, in automatic target recognition systems, in the organization of secure communication channels. The military sphere, as a rule, stimulates scientific and technological progress, and therefore further growth in the number of solutions using ML in defense systems is expected.

Most modern ML-built solutions are based on Artificial Neural Networks (ANN). It is the models built on ANN that today provide high accuracy of pattern recognition, decision-making based on large data sets, automatic classification and prediction of system behavior. However, like any complex technology, ANN has a number of limitations, imposed by complexity of interpretation, possibility of overfitting, sensitivity to the choice of the training set, as well as vulnerabilities to external inter­ference, for example, to attacking effects on input data.

When using ML-based systems, situations related to errors in the functional design of the system often arise. There is a known case when
a Chinese video surveillance system using ML for face recognition repeatedly recorded traffic violations by a Chinese woman whose face was depicted on the side of a moving bus, which led to a decrease in her social rating, adopted in China. Similar errors can lead to serious consequences in defense systems, where an identification error can mean, for example, incorrect classification of an object as hostile or, conversely, harmless.

In this regard, one of the important tasks is the development of systems based on machine learning that have known beforehand limitations and are resistant to possible data distortions. This is especially relevant in the context of Intrusion Detection Systems (IDS), where the ability to promptly detect malware activity in information networks depends on the efficiency of the built-in neural networks¹. When analyzing the efficiency of performance of deterministic IDS, such metrics as the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) are traditionally used. In conditions where an IDS with ML-based modules or an IDS built entirely on neural networks is considered, the question of the stability of the neural network model itself arises due to the fact that the training set itself can be partially or completely compro­mised using various methods or techniques. In this case, it becomes criti­cally important to determine both the metrics themselves that characterize the ML model and the deviation of their values ​​from the reference ones.

In this study, we consider a model for assessing the resilience of a Ge­ne­rative Adversarial Network (GAN*) to fuzzing**. GAN is integrated into the Snort IDS. GAN was chosen due to its ability to not only generate syn­thet­ic datasets, but also to classify based on a built-in discriminator. Due to its architecture, GAN is increasingly used in recognition and classifi­ca­tion tasks. Researchers of ML-based systems solve the classification prob­lem based on known and/or developed algorithms (within the specified con­straints)², but assessing the resilience of GAN based on traffic and a training set using fuzzing has not been considered. Accuracy, precision, and recall*** were chosen as the parameters to be assessed when input data is distorted. The choice of these parameters out of the multitude of those characterizing neural networks is due to their complementarity (within the framework of a multi-class classification), which allows for a more accurate verification of the results.

To conduct the experiment, a model was developed that integrated network security tools, neural network analysis, and fuzzing data distortion.

The scientific novelty lies in the construction of a model for estimating the deviation of the given GAN parameters in the presence of fuzzing distortions.

Statement of the problem:

the study of the model for assessing the deviation of the given metrics from the reference value in the presence of fuzzing distortions.

Boundary conditions:

• in the study, exclusively open, public datasets were used,
• the model was trained using the “unsupervised learning” method,

the neural network was trained with fixed parameters without hyper­parameter optimization. Hyperparameter optimization would undoubtedly improve the model’s performance. However, the primary objective of this study was to assess the stability of the GAN architecture under fuzzing-induced distortions using fixed parameters. This constraint was introduced deliberately in order to observe the model’s natural behavior without the influence of optimization techniques,

• the stability assessment was carried out at 6 levels of distortions of the training set: 0 % – control, 5 %, 10 %, 20 %, 30 %, 50 %.

Description of the experiment

The Windows Server 2019 operating system (OS) with the activated role of the Hyper-V virtualization is installed in a virtual environment based on a High-Performance Computing Cluster (HPCC). A Software-Defined Networking (SDN) is configured in the virtual environment, in which the Snort IDS with a GAN-based plug-in module is deployed. An FTP (File Transfer Protocol, FTP) server and Kali Linux OS with Metasploit software are also installed. The IDS protects the FTP server deployed on Windows Server 2019 OS, which imitates the Infrastructure. The architecture of building the Infrastructure in SDN is shown in Fig. 1.

Using Metasploit software from the Kali Linux distribution, an attack is carried out on the IDS using malware (abc, cheeba, december_3, stasi, otario, dm, v-sign, tequila, flip) together with a dataset generated by the software fuzzers Boofuzz ​​ (for fuzzing network requests at the HTTP, FTP, DNS) and Radamsa (for generating mutations at the level of a raw packet and a pcap file)³. Some solutions obtained in studies were also used⁴. For each level, GAN training, integration of the trained model into the IDS and collection of the specified metrics were carried out.

The caused distortions included the following options:

• injection of random bytes,
• repetition of key sections of the packet,
• deletion of fields,
• change of the order of headers,
• insertion of logically incorrect values ​​(for example, packet size <0).

 

Fig. 1. Architecture of building Infrastructure in SDN

Նկ. 1. ԾՊՑ-ում Ենթակառուցվածքի կառուցավորման ճարտարապետությունը

1. FTP Server (ՆՓՀ-սպասարկիչ)
2. Private, Vlan ID=100 (Մասնավոր, ԵՏՀՑ, Նույնացուցիչ=100)
3. Snort&GAN module (NAT) («Սնորտ»&ԳՄՑ մոդուլ (ՑՀՀ))
4. Private, Vlan ID=110 (Մասնավոր, ԵՏՀՑ Նույնացուցիչ=110)
5. SDN base on Hyper-V (ԾՊՑ հիմք «Հայպեր-Վի»-ի վրա)
6. Private, Vlan ID=110 (Մասնավոր, ԵՏՀՑ Նույնացուցիչ=110)
7. Kali Linux (Metasploit) «Քալի Լինուքս» («Մետասպլոյտ»)

 

The data distortion methods were selected based on both specialized literature and the analysis of real-world network attacks. The selection of these methods was driven by their practical applicability within the fuzzing framework and their potential to influence the classification accuracy of pre-trained data.

To integrate GAN into IDS, the developed scripts fazzing_gan.py, classify_gan.py (Fig. 2) were used.

In active mode, the corresponding IDS preprocessor passes the network packet to an external Python program, where it is processed by the trained GAN and classified as «malware» or «normal». The result is returned back to the IDS, where it affects the triggering of the rule. Part of the program code for integrating GAN into IDS (Snort) is as follows:alert tcp any any -> any 80 (msg:”Check with GAN”; content:”GET”; sid: 1000001; classtype:policy-violation; program:”/usr/local/bin/classify_ gan.py”;)

It is important to note that a number of other researchers have also addressed this problem, particularly by employing autoencoders, multilayer perceptrons, and classical machine learning algorithms such as decision trees and support vector machines (SVM). Experimental comparisons indicate that while autoencoders can achieve high classification accuracy under conditions of low data distortion, their performance-especially in terms of recall-tends to degrade significantly when exposed to intensive fuzzing attacks. This underlines the comparative advantage of GAN-based models in scenarios involving deliberate input perturbations, such as targeted network attacks. At the same time, approaches based on decision trees and SVMs typically require large-scale and well-verified training datasets, which may not always be feasible in practical applications.

Fig. 2. Integration of GAN into IDS

Նկ. 2. ԳՄՑ ինտեգրում ՆՀՀ-ին

Results of the study

The Table 1 presents the results of the study on the model for assessing the degree of deviation of the accuracy, precision, recall metrics from the reference value in three training epochs. Visualization of the change in the accuracy, precision, recall parameters in three training epochs is shown in Fig. 3. Visualization was performed using TensorFlow software.

Table 1

Results of the model research for III training epoch

Distortion level (%)	Accuracy (%)			Precision (%)			Recall (%)
	training epoch
	I	II	III	I	II	III	I	II	III
0	93.8	94.2	96.2	90.1	92.4	95.4	92.2	96.3	97.0
5	91.4	90.2	94.8	87.0	89.8	92.3	90.5	92.4	94.6
10	87.2	90.6	91.5	85.1	86.3	88.6	88.1	90.4	91.2
20	82.0	85.3	87.1	78.2	78.6	81.2	80.4	83.6	84.8
30	76.7	78.3	82.4	70.6	72.5	74.5	72.6	76.2	78.0
50	70.3	76.2	82.4	55.4	58.6	63.1	62.3	64.3	66.5

Training was performed at four epochs, but upon reaching the fourth training epoch, overfitting of the network occurred, as a result of which the experiment was not continued.

Fig. 3. Visualization of changes in the parameters accuracy, precision, recall
for three training epochs

Նկ. 3. ՈՒսուցման երեք ժամանակաշրջանների դեպքում «ճշտութ­յուն», «ճշգրտություն», «զգայունություն» պարամետրների փոփոխման տեսանելիացում

Additional training (up to three epochs) significantly improves the model’s resilience to fuzzing biases, making this strategy particularly valuable for systems operating in hostile or unstable network conditions.

Conclusions

The carried out study leads to the following conclusions:

Accuracy

• accuracy decreases as the level of data distortion increases,
• the best results are achieved in the third training epoch, which indicates the effectiveness of additional training,
• a particularly noticeable increase in accuracy is observed after three training epochs. The model remains significantly more stable compared to the first and second epochs.

Precision

• precision also shows an improvement from epoch I to III, but it is more sensitive to the increase in distortion,
• at low distortions (0 – 10 %), the increase in precision between epochs is moderately expressed, while at 30 – 50 % of values it becomes critical.

Recall

• recall behavior is more stable than precision, and shows good growth from epoch to epoch at all distortion levels,
• it is especially important that even at 50 % of distortions, after epoch III recall exceeds 66 %, which is significantly better compared to epoch I (62,3 %).

Since the recall parameter reflects the ability to find all real threats, reducing this parameter is especially critical in security-related systems.

Conclusion

Artificial intelligence models, especially in the field of security, must be robust not only to classic errors, but also to deliberate fuzzing bias. This study demonstrates that even a small percentage of biases can lead to significant classification failures. This highlights the need for regular robustness testing, assessment of trust in training data sources, and design of “fuzz-resistant” architectures.

The conducted study showed that the robustness of a neural network model significantly depends on the quality of the training data. GAN, despite its generative potential, loses the ability to extract stable patterns from distorted data and begins to reproduce the “wrong norm”. Without additional mechanisms for protecting against biases (autoencoder filtering, input data trust control) neural network models are vulnerable in case of attacks based on distorted data, which leads to incorrect operation of systems based on them.

Translated by Narine Mkrtchyan

¹ Տես / See Saeid Jamshidi, Kawser Wazed Nafi, Amin Nikanjam, Foutse Khomh. Evaluating Machine Learning-Driven Intrusion Detection Systems in IoT: Performance and Energy Consumption (https://doi.org/10.48550/arXiv.2504.09634); Ahnaf Akif. Binary and Multi-Class Intrusion Detection in IoT Using Standalone and Hybrid Machine and Deep Learning Models, December 2024 (https://doi.org/10.48550/arXiv. 2503.22684); Tasnimul Hasan, Abrar Hossain, Mufakir Qamar Ansari, Talha Hussain Syed. Enhanced Intrusion Detection in IIoT Networks: A Lightweight Approach with Autoencoder-Based Feature Learning, 25 January 2025 (https://doi.org/10.48550/ arXiv.2501.15266); Muhammad Zawad Mahmud, Shahran Rahman Alve, Samiha Islam, Mohammad Monirujjaman Khan. SDN Intrusion Detection Using Machine Learning Method (https://doi.org/10.48550/arXiv.2411.05888):

2 Տես / See “Scikit-learn: Machine learning in Python” (https://scikit-learn.org/stable/ index.html); Varij Saini, Rudraksh Gupta, Neel Soni. OpCode-Based Malware Classi­fi­ca­tion Using Machine Learning and Deep Learning Techniques, 18 April 2025 (https://doi.org/10.48550/arXiv.2504.13408); Samuel Ward, Alain Zemkoho, Selin Ahipasaoglu. Mathematical programs with complementarity constraints and application to hyperparameter tuning for nonlinear support vector machines, 17 April 2025 (https:// doi.org/10.48550/arXiv.2504.13006):

3 Տես / See “Malware database” (http://vxvault.net/ViriList.php); “Malware reposi­tory” (https://avcaesar.malware.lu/); “Viruses repository” (https://virusshare.com/); “Boofuzz” նախագծի պաշտոնական կայքէջը / Official web-page of the “Boofuzz” project (https://boofuzz.readthedocs.io/en/stable/); “Radamsa” նախագծի պաշ­տոնա­կան կայքէջը / Official web-page of the “Radamsa” project (https://gitlab.com/akihe/ radamsa):

4 Տես / See Sevak Sargsyan, Jivan Hakobyan, Matevos Mehrabyan, Maxim Mishechkin, Vitaliy Akozin, Shamil Kurmangaleev. ISP-Fuzzer: Extendable Fuzzing Framework, 24 October 2019 (https://doi.org/10.1109/IVMEM.2019.00017); Sevak Sargsyan, Lyudovikos Nersisyan, Karen Sargsyan, Vahagn Melkonyan. Improving fuzzing efficiency based on extracted constant values, 22 March 2023 (https://doi.org/ 10.1109/ISPRAS57371.2022.10076849):

* GAN is an artificial neural network consisting of two models (generator and discriminator) that learn simultaneously during the competition:

• generator (G) – creates synthetic data, simulating real data,
• discriminator (D) – tries to distinguish real data from synthetic data (see Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio. Generative Adversarial Networks, 10 June 2014 (https://doi.org/10.48550/arXiv.1406.2661)).

** Fuzzing is a data generation technique aimed at detecting incorrect system behavior (see Е. А. Ефремов, А. Е. Ковалевский. Фаззинг. Методы и средства фаззинга. «Между­народный журнал гуманитарных и естественных наук», 2016, Т. 1, № 8 (https:// cyberleninka.ru/article/n/fazzing-metody-i-sredstva-fazzinga)).

*** Accuracy – the proportion of correct classifications.

Precision – the accuracy of positive classification.

Recall – the completeness of detection of malicious traffic (see “Encyclopedia of Machine Learning”. Ed. by Claude Sammut, Geoffrey I. Webb. New York, 2011 (https:// pzs.dstu.dp.ua/DataMining/bibl/Encyclopedia%20Machine%20Learning%202011.pdf);

Թ. Վ. Ջամ­ղարյան, Թ. Ն. Շահնազարյան, Պաշտպանական ոլորտում կեղծ ցանցային ենթա­­կա­ռուցվածքում նեյրոնային ցանցի կիրառման մոդելի հետա­զոտություն, T. V. Jamghar­yan, T. N. Shahnazaryan. A study of a model of neural network application in the decoy infrastructure in the defense sphere (in Arm. lang.). “Haikakan Banak”, 2024, N2 (http://haykakanbanak.am/wp-content/uploads/2024/09/HB-2-Union-10-Kazm-Text-plus-Nerdir-Stugel.pdf)):