M.S. Candidate: Serbay Çetinkaya
Program: Cybersecurity
Date: 13.11.2026 / 11:00
Place: B-116

Abstract: The Domain Name System (DNS) is a fundamental component of internet communication, translating human-readable domain names into IP addresses. However, queries in DNS are transmitted in plaintext, which makes it vulnerable to eavesdropping, manipulation and censorship. Encrypted DNS protocols such as DNS over HTTPS (DoH) have been introduced to enhance privacy and security. While DoH improves user privacy by encrypting DNS queries within HTTPS traffic, it also brings challenges for network monitoring, as Cyber Threat Actors (CTA) exploit DoH to evade security mechanisms. Traditional monitoring techniques relying on packet inspection and signature-based detection become less effective due to encryption. This thesis proposes an autoencoder based anomaly detection framework for DoH traffic, utilizing the BCCC-CIRA-CIC-DoHBrw-2020 dataset. Unlike supervised learning models that depend on labeled datasets, our approach uses unsupervised learning to detect deviations from normal DoH traffic patterns. The autoencoder is trained to reconstruct benign traffic and anomalies are identified based on reconstruction errors. The model improves detection accuracy by using a balanced dataset and also distinguishes between normal and potentially malicious DoH activity. Additionally, our framework is designed for real-time detection which allows proactive identification of DoH attacks without compromising user privacy. The final model reached a malicious recall of 0.99 and an overall accuracy of 0.945 with a benign recall of 0.64. The results demonstrate that autoencoder based anomaly detection provides an adaptable real-time solution for monitoring encrypted DNS traffic.