AI-Driven Phishing Efficiency Exacerbates Human Vigilance Deficits

The rapid integration of artificial intelligence is intensifying a fundamental weakness in cybersecurity: the natural decay of human attention, known as the vigilance decrement. As AI-driven tools increase the volume and sophistication of digital interactions, security professionals are increasingly tasked with monitoring high-velocity environments that exceed human cognitive limits.

This shift is occurring alongside a fundamental change in how social engineering attacks are executed. Traditionally, security training focused on identifying “tells,” such as poor grammar or suspicious formatting in phishing emails. However, AI is effectively erasing these indicators.

The Vanishing Phishing “Tell”

Research conducted by Bruce Schneier and Fredrik Heiding indicates that AI can reduce the cost of phishing campaigns by more than 95 per cent. Crucially, these AI-generated lures can match the click-through rates of highly skilled human teams, making them nearly indistinguishable from legitimate communications.

This capability creates a dual pressure on workers. Not only is the quality of the threat increasing, but the sheer volume of digital “aircraft” to monitor is growing. As knowledge workers manage multiple automated agents and rapid-fire communications, the likelihood of missing a single, high-consequence threat increases.

The High Cost of Business Email Compromise

The financial implications of this attention deficit are most visible in business email compromise (BEC) attacks. While these incidents represent a relatively small fraction of total cyber complaints, they account for a disproportionate amount of financial loss.

In 2024, the FBI recorded 21,442 BEC complaints. While these were part of a much larger pool of over 850,000 total complaints, BEC was responsible for $2.77 billion of the $16.6 billion in reported losses for the year. These attacks are particularly damaging because they lead to irreversible actions, such as unauthorised wire transfers, changes to banking details, or credential resets.

Shifting to Action-Based Controls

To counter these risks, security experts suggest a model based on high-stakes industries like aviation. In aviation, when human controllers and automated systems disagree, the protocol is to follow the machine. However, this relies on machines that are “dumb” and certain, rather than machines that attempt to guess intent.

In a cybersecurity context, this means moving away from relying on a human’s ability to recognise a fraudulent message and instead implementing controls that fire on the action itself. Effective mitigations include:

• Mandatory callbacks: Requiring a phone call to a known, verified number for any change in banking instructions.
• Dual control: Requiring two separate authorised individuals to approve high-value wire transfers.
• Automated thresholds: Implementing hard limits that automatically halt payments that exceed specific risk parameters.

While AI-driven security tools that attempt to judge intent are being developed, they face a significant hurdle in reliability. A system that produces too many false positives may be bypassed by treasury departments, while a system that is too quiet may fail to stop a sophisticated adversary. Until intent-based AI can prove it is as reliable as the “dumb” automated controls used in aviation, the focus must remain on securing the critical moments where identity is surrendered or money is moved.