Who is on the other end? AI can already seem human and copy a voice

Imagine this. Your daughter calls you directly. You recognise her voice. She is nervous, has a problem and needs you to transfer some money. You talk to her, ask what happened, and she answers. Everything seems to add up.

What evidence would you need to believe it is her?

For a long time, hearing a familiar voice has helped put doubts to rest. Today, we have reasons to ask for more. Some systems can generate conversations that seem human, and some tools can produce speech that sounds like a particular person from a short recording.[7][3][4]

To prepare this article, I have tried free voice cloning options available today. The results are quite good. That is precisely why I find them worrying.

A conversation can build trust. A voice can reinforce it. Urgency can make us act before checking who is asking for what.

The deception needs to last long enough to secure a decision. A bank transfer. A password. A document you should never have sent.

Ferrari: one question interrupted the deception

In July 2024, a Ferrari executive received WhatsApp messages from someone posing as Benedetto Vigna, the CEO. The messages mentioned a confidential acquisition, asked for discretion and said documents would follow for the executive to sign. Then came a call. According to Bloomberg's reporting, republished by The Spokesman-Review, the voice even imitated Vigna's southern Italian accent.[11]

The supposed boss explained that he was using another phone and asked for help with a currency hedging transaction. The executive grew suspicious. The request seemed odd, and he noticed slightly mechanical intonations. So he asked a question: what book had Vigna recommended to him a few days earlier? The call ended.[11]

The executive stopped accepting the voice as proof of identity and asked for an additional check. That ended the attempt.

Arup: the video call that led to fifteen transfers

In another case in 2024, an employee in Arup's finance department in Hong Kong received a request to carry out a secret transaction. At first, he was suspicious. Then he joined a video call in which he believed he recognised the chief financial officer and other colleagues. Their faces and voices made him set aside his doubts, according to the police account reported by CNN.[12]

The employee made fifteen transfers totalling HK$200 million, around US$25.6 million. Arup confirmed to CNN that it had suffered the fraud and that fake voices and images had been used.[12]

Here, the false identity had company. The meeting seemed to provide several confirmations: the boss, the colleagues, their faces, their voices. But all were part of the same staged scene.

These two cases explain why my own tests worry me. At Ferrari, verification stopped the attempt. At Arup, recognising the people on screen was enough to turn an initial suspicion into a multimillion-dollar loss.

Five minutes to seem human

Cameron Jones and Benjamin Bergen, researchers at the University of California San Diego, tested something very specific: whether participants could distinguish an AI from a human in a written conversation. In two randomised, preregistered experiments, each evaluator held two five-minute conversations at the same time: one with another person and one with an artificial system. They then chose which one they thought was human.[7]

When GPT-4.5 was instructed to adopt a human persona, evaluators chose it as human 73% of the time. Under that condition, they mistook the machine for the human more often than they correctly identified the real person.[7]

In five minutes of written conversation, the machine managed to seem more human than the human. Natural writing no longer serves as proof that there is a person behind it.

If we expect to spot a bot through poor writing, stiff phrasing or its failure to understand a joke, we are relying on a test that can fail. The experiment's result means we need to take that possibility seriously.

Persuading you is a different ability

A machine can seem human without persuading you of anything. And it can persuade you even when you know perfectly well that it is a machine.

Francesco Salvi and his colleagues studied persuasion in debates involving 900 participants. They compared human interlocutors and GPT-4, with and without access to sociodemographic information about the person they were debating. They measured agreement with a position before and after the conversation.[1]

In comparisons that were not tied, GPT-4 with personal information was more persuasive than humans 64.4% of the time. The measure was the change in agreement with a position after the debate.[1]

A later correction retains the advantage over the human baseline, while leaving the difference between GPT-4 with and without personalisation inconclusive.[6]

For someone taking part in a chat, the practical consequence is simple: a good argument deserves examination. The identity of the person presenting it needs a separate check.

When the experience you are told about is invented

The University of Zurich case adds another dimension. According to The Decoder's reconstruction, based on project documents and comments released by moderators, researchers introduced accounts posting AI-generated replies into Reddit's r/ChangeMyView forum without informing its users.[8]

The team posted 1,783 comments and received 137 “deltas”: the recognition a user gives when a contribution changes or substantially challenges their position. Some accounts used invented identities and personal experiences to give their arguments credibility or emotional weight. The researchers eventually announced that they would not publish the study.[8]

Those accounts owed some of their credibility to lives nobody had lived.

“This happened to me” is no ordinary phrase. It asks you to credit the speaker with an experience. “I work with victims” asks you to recognise expertise. If those biographies are false, the deception exists even if the argument includes accurate facts.

There is a difference between helping someone write better and inventing the person who is supposedly writing.

Your voice can also stop being proof

I wanted to try that other part of the problem myself. For this article, I have used free voice cloning options, and I have found the results quite good. Good enough to take their potential to deceive very seriously.

The technical groundwork has been public for some time. In March 2024, OpenAI introduced Voice Engine, which generated speech resembling a person's voice from a fifteen-second sample, in a restricted-access trial.[3]

Microsoft had demonstrated personalised speech synthesis with VALL-E using three seconds of reference audio. Its own documentation warns that similarity and naturalness depend on factors such as the audio's quality and length, and background noise.[4]

Today, there are also open models such as VoxCPM2: its code and weights are released under the Apache 2.0 licence, and it offers cloning from a short sample. It can run on a compatible computer without paying a provider for each generation.[9]

Think about how that result could be used: a short audio clip, a phone, an urgent request. The voice comes with a story and a reason to act quickly.

“I would notice” is not enough of a safeguard

Kimberly Mai and her colleagues presented real and synthetic voices to 529 participants, in English and Mandarin. In the condition where audio clips were assessed individually, listeners correctly identified 73% of the fake recordings. Familiarising them with examples beforehand improved the results only slightly.[2]

The task was to classify recordings, and participants did better when they could compare them in pairs.[2] In an everyday conversation, we hear the audio as part of a story, with our own expectations and worries. I would not stake the security of a payment on my ability to detect a fake voice.

That is why I would not make security depend on spotting an odd breath, an artificial intonation or a small glitch. Those clues can raise suspicions. Their absence leaves the recording's authenticity unverified.

The story you hear can matter as much as the voice

A preprint by Fred Heiding and his colleagues looks more directly at AI phone fraud. Their experimental survey included 4,100 US adults, who were exposed to recordings or transcripts of different scam scenarios, and was supplemented with interviews.[5]

Across the scenarios, 16.5% said they would comply with the request or expressed uncertainty. In the category involving a family member in distress, the proportion reached approximately 36%. The survey measured willingness to comply with the request, including uncertain responses.[5]

The authors identified the interlocutor's persuasiveness as the strongest predictor of compliance.[5]

We can become so focused on whether the voice sounds exactly right that we forget what we are being asked to do, why it has to happen now and how we can check it.

The audio is one part. The request, the context and the pressure are the others.

In a chat, a crowd can be staged too

Beyond money, there is also the shaping of opinion. A profile that seems human can join a discussion. Several profiles can create an appearance of agreement, rejection or shared experience. We may encounter a staged conversation and interpret it as a spontaneous reaction.

The precaution I suggest is to separate the apparent number of voices from the quality of the evidence. Ten accounts repeating an accusation are not ten independent verifications. A moving biography does not replace a source. An elaborate answer does not turn a claim into a fact either.

That holds even if all the accounts are human. AI gives us one more reason to keep that standard.

Verification has to happen outside the scene

The rule I want to remember is this: recognising a voice is not the same as verifying an identity. And verifying an identity does not authorise every request.

My practical conclusion is to stop asking the same message to prove its own authenticity.

If you receive an urgent call, hang up and call back using the number you already had saved. If everything comes through a chat—messages, audio clips or a video call—check the request outside that thread before acting.

• Go back to the contact you already knew. Call the number you had saved, not the one that has just appeared or one supplied by the suspicious message.
• Separate urgency from authorisation. An emergency may require speed, but it does not make a request authentic by itself. Check with another family member, manager or regular contact.
• In a company, stick to the procedure. An audio clip attributed to the director should not be enough to change a bank account, order a payment or hand over confidential information. Use the internal verification channel and the required approvals.
• Do not treat personal information as a password. Knowing your name, where you work or something you have shared publicly does not establish a relationship with you.
• If you have already sent money or information, act. Contact your bank or the person responsible for the affected service immediately, keep the messages and report what happened to the relevant authorities.

Useful AI does not need a false identity

A synthetic voice can help people communicate. AI can translate, explain, write or prepare an argument that a person then reviews and takes responsibility for. That work does not need an invented victim, family member or professional to get a hearing.

I do not want to give up the tools we are using to build useful things. I want to know who is responsible for what I receive and what evidence I have before I act.

We will have fewer and fewer reasons to accept “sounds human” as verification. And more reasons to ask a specific question:

How do I know it is you?

Origins and sources

The starting point was a video by Lógica Globalizada about artificial identities and trust on the internet.[10] The work of Jones and Bergen, Salvi and colleagues, and Mai and colleagues allows us to examine the ability to seem human, persuasion and synthetic voice detection separately. The Zurich case is covered through journalistic reporting on an unpublished study; the work by Heiding and colleagues is available as a preprint. The tests with free voice cloning options are the author's personal experience.