2.37 Million Emails Per Day: Phishing Campaign Exploits AI 'Jailbreak' Trick

Microsoft has identified an AI 'jailbreak' trick used in a massive email phishing campaign that peaked at over 2.37 million emails per day. The technique, known as ASCII smuggling, uses invisible Unicode characters to evade filters and trick email systems into delivering the messages.

The campaign, which started on February 9 and lasted for roughly three months, used ActiveCampaign's marketing automation platform to send out millions of emails with finance-themed lures. Most of these emails contained hidden tag characters between words like 'funding', making it difficult for filters to detect them.

Microsoft notes that the technique was originally developed by AI security researchers to expose blind spots in models, but has now been repurposed for malicious purposes. The company's own Defender for Office 365 stack still managed to flag over 99% of the messages through layers that didn't depend on catching the tag characters directly.

Microsoft warns that this technique is a cautionary tale about how security research can be misused, and emphasizes the importance of normalizing text before matching it with filters, as well as testing whether they treat different versions of the same word equally.