Social media platforms have become valuable sources for detecting phishing due to their widespread use and rapid content sharing. Among them, Telegram stands out as an open, data-rich environment, offering access to large public groups where phishing links frequently appear. This study investigates whether Large Language Models (LLMs) can serve as effective tools for phishing detection on Telegram. We systematically evaluate three representative LLMs — GPT-4o, LLaMA-3.1, and DeepSeek V3 — across multiple prompting strategies and input configurations. For comparison, we employ an improved version of traditional feature-based machine learning models from the literature, enhanced with resampling techniques to address the strong class imbalance in our data. Experimental results show that while LLMs demonstrate some potential, achieving a Recall for the phishing class of up to 0.554 with DeepSeek — representing the model’s ability to correctly identify 55.4% of all actual phishing messages — they suffer from extremely low precision (0.065 with DeepSeek), resulting in a low F1-score of 0.114. In contrast, our feature-based models achieve a significantly more reliable precision-recall balance, with an overall phishing F1-score of 0.749 and a precision of 0.738. The findings in this study highlight the potential and current limitations of LLMs for phishing detection in dynamic instant messaging environments and emphasize the superior performance of platform-tailored models.
Md Erfan, Paula Branco, Guy-Vincent Jourdan· Digital Threats: Research an...· 0 citations
Connected and Autonomous Vehicles (CAVs) rely on interconnected software and hardware components, including sensors, Electronic Control Units, in-vehicle infotainment systems, and telematics units, where vulnerabilities can compromise assets, users, and vehicle operations. These vulnerabilities are commonly documented as plain text in the Common Vulnerabilities and Exposures (CVE) database; however, security practitioners require structured information about affected assets, types of weaknesses, and attack behaviors to effectively mitigate the risks from these vulnerabilities. To this end, we evaluate open-weight Large Language Models (LLMs) for generating Structured Threat Information Expression (STIX), a well-known structured format for representing threat information, for CAV-related CVEs. We construct a dataset called CAV-STIXGen that maps CAV vulnerability descriptions to STIX domain objects (SDO), STIX relationship objects (SRO), Common Weakness Enumeration (CWE), and MITRE ATT&CK techniques mappings. Using this dataset, we evaluated 11 open-weight LLMs (4B to 120B parameters) across various prompting strategies and temperatures. Single-model configurations achieve F1 scores of 0.94 for SDO, 0.63 for SRO, and 0.99 for CWE mapping, while complete MITRE ATT&CK mapping remains challenging. In a multi-agent setup, Gemma-4-31B and Codestral-22B achieve F1 scores of 0.91 for SDOs and 0.43 for SROs, respectively. Lastly, we analyze CWE and MITRE ATT&CK co-occurrences to identify recurring threat patterns in the CAV domain, demonstrating how AI-assisted vulnerability-to-STIX translation can automate threat intelligence and prioritize defense in transportation security.
Md Erfan, A. Ryan, Md. Kamal Hossain Chowdhury et al.· 0 citations