Synthetic Celebs & Fabricated Family: Protecting Against a New Breed of Cybercrime

Recent technological developments mean that AI-generated speech has skyrocketed in quality, to the point where it is in some cases virtually indistinguishable from genuine human speech. Even more concerningly, speech can be specially engineered to sound like a specific speaker, a phenomenon known as voice spoofing. Spoofs (also known as voice clones or auditory deepfakes) can fool automatic speaker verification (ASV) systems used in secure settings such as banking and government, potentially granting fraudsters access.

Human listeners are similarly at risk. To best illustrate this, we turn now to the ‘the 3 Ds of Deception’:

Disinform - in which spoofs are used to instigate and disseminate knowingly false information. In successful cases, listeners believe and further spread the information, because to all appearances, the source is an influential, ostensibly trustworthy figure. This D is most commonly seen in politics, where cybercriminals impersonate people in power to suppress voters or manipulate elections. At their worst, spoofs of this nature have the potential to threaten democracy itself.

Defame – unlike the other two Ds, the main victims here are those being spoofed, not the listeners being deceived. Defamatory voice spoofs create fake audio of their victim saying something offensive, inflammatory, or illegal, triggering social and sometimes financial backlash. While high-profile cases exist, the risk to ordinary people can be greater. A celebrity might weather the backlash; a working-class shop assistant might not. A deepfake of the latter making a racist remark would not spark the same levels of international outrage as, say, one of Billie Eilish would. However, if both lost their jobs as a result, one would lose their only income, while the other would still be worth 
$50 million.

Defraud – voice spoofs can be used to impersonate someone with a relationship to a listener, with the aim of extracting money from that listener. AI-mediated fraud is a new type of confidence scam, exploiting pre-existing relationships and trust, rather than building trust over time. Fraud can involve false celebrity endorsements or phone scams, in which an imposter poses as a colleague, friend, or family member in financial need. This D has already shown to be hugely damaging, with millions of pounds being stolen this way in recent years (see Stupp, 2019; Brewster, 2023; Bunn, 2023; Verma, 2023).

The conditions dictating accurate discrimination between genuine and synthetic speech are little understood.

Presently, attempting to protect human listeners from AI-mediated deception is difficult, since the conditions dictating accurate discrimination between genuine and synthetic speech are little understood. There is, however, one factor that may prove invaluable to increasing this understanding: familiarity. As demonstrated above by all 3 Ds, spoofed voices are celebrities, colleagues, family members – people who are known to the listener. Listeners will have a developed knowledge base of the spoofed speaker’s genuine voice. Surely, then, this familiarity will make listeners better at recognising when the speaker is being impersonated?

Our preliminary findings investigating the influence of familiarity on detection of deepfake voices suggest that this is not the case. Familiarity may degrade performance. Of 120 participants studied, it was found that accurate identification of AI-generated samples dropped to 63% if the perceived speaker was a very well-known celebrity (Donald Trump), compared to one with a more limited fanbase (Linus Sebastian), where accuracy rates were around 73%.

One possible explanation for this is that familiarity with someone’s voice isn’t limited to just knowing how they sound. The voice activates a wealth of rich information about that person: knowledge of their experiences, personalities, values, and relationships. In the study above, many listeners left comments indicating strong negative feelings for Donald Trump. One listener described the speaker they were listening to as an ‘idiot’, and therefore decided it must genuinely be Trump, rather than a spoof. In short, they used a highly subjective value judgement to discriminate between human and AI-generated speech, rather than the voice alone.

This suggests that a listener’s personal experience with the speaker might overshadow objectivity in decision-making. Compare this to a stranger’s voice – the listener has had no past contact with them, and so they can assess the voice impartially.

Future work in this research series will diverge from familiar-celebrity spoofs in favour of spoof detection when there are familiar-intimate relationships between listener & perceived speaker (i.e., family, friends). From the insights gained – particularly why listeners make their decisions – we can explore the cues that are being registered, and more crucially those that are being missed. Spoofing is not an isolated threat; people and organisations at all levels of society are potential victims. Understanding how and why people are deceived is essential if we wish to develop our resilience against them.

Download the article

Hope McVean is a current PhD student at Lancaster University, working primarily in FACTOR (Forensic Linguistics, Cybersecurity and Technology Research).