What matters in AI.

Subscribe

Learn / AI basics

Definition · AI basics

Voice cloning

Voice cloning is the use of AI to make new speech in the voice of one actual person. The software learns the voice from a recorded sample. Then it can say each text that you give it in that voice. The research system VALL-E (2023) used a 3-second recording. The FBI says that vishing can use AI-generated voices.

Last reviewed

Key points

  • Voice cloning copies the sound of the voice of one person from a sample.
  • The sample can be very short. VALL-E (January 2023) used a 3-second recording of a new speaker.
  • In February 2023, a reporter used a clone of his voice to go through the voice ID of his bank.
  • In 2024, the FCC fined the maker of a fake Biden robocall $6 million.
  • Do not trust a voice alone. Find the number of the person on your own and phone back.

How voice cloning works

Voice cloning software has two inputs: a sample of the voice and a text. The output is new speech in that voice.

One frequent design has three parts (Jia and others, 2018):

  1. Speaker encoder. It hears “seconds of reference speech” and makes a short numeric record of the voice, an embedding.
  2. Synthesizer. It changes the text into a spectrogram in that voice.
  3. Vocoder. It changes the spectrogram into audio.
A voice sample and a text go in. Speech in the cloned voice comes out.
A voice sample and a text go in. Speech in the cloned voice comes out.

Newer systems work like a language model. VALL-E learned from 60,000 hours of English speech. It copied a new speaker from a 3-second recording.

Why voice cloning matters

Persons and systems trust a known voice. Voice cloning breaks that trust at low cost.

  • Voice ID. In February 2023, a Vice reporter made a clone from about five minutes of his speech. The clone opened his Lloyds Bank account.
  • Vishing. In May 2025, the FBI warned of AI voice messages that impersonate senior US officials. This is a type of phishing.
  • Elections. On 21 January 2024, 9,581 calls with a fake Biden voice told New Hampshire voters not to vote.

A cloned voice is the audio type of a deepfake.

In practice

Instant clone vs professional clone

ElevenLabs, a vendor, gives two levels:

Method How it learns the voice Audio (ElevenLabs docs)
Instant clone Uses the sample at the time of speech. The model does not change. Less than 2 minutes
Professional clone Fine-tuning of the model on the samples About 30 minutes

What US regulators did, 2024 to 2025

Date Body Step
8 Feb 2024 FCC Ruled that AI voices, such as voice cloning, are “artificial” voices under the TCPA. Calls need prior express consent.
Apr 2024 FTC Named the winners of its Voice Cloning Challenge. Three winners share US $35,000.
26 Sep 2024 FCC Adopted an order that fined Steve Kramer $6,000,000 for the Biden calls (Truth in Caller ID Act).
15 May 2025 FBI Warned of AI voice messages that impersonate senior US officials.

In March 2025, Consumer Reports tested six voice cloning products: Descript, ElevenLabs, Lovo, PlayHT, Resemble AI and Speechify. The Register reports that four of them used a checkbox only. The user stated a legal right to clone the voice.

3 types of defence from the FTC challenge

IEEE Spectrum reports on the winners of the FTC Voice Cloning Challenge (April 2024). They show three types of defence:

  1. Change the sample. DeFake adds small changes to voice recordings. Then AI models have a harder time to clone them. IEEE Spectrum says that its efficiency must improve.
  2. Prove a human source. OriginStory uses a microphone with sensors for body signals of speech. It adds a verification mark to the recording. See AI output watermarking.
  3. Find the clone. AI Detect flags fake voices in real time. Pindrop examines speech in 2-second intervals. See deepfake detection.

What a security team can do

  • Do not use a voice alone as proof of identity. The Lloyds test shows that a clone can go through voice ID.
  • For payments and access changes, find a number for the person on your own and phone back. The FBI tells persons to make sure of the identity of the caller.
  • Tell staff that a known voice is not proof.

Trade-offs

  • Less audio, more risk. VALL-E used a 3-second recording. A short sample is sufficient for some systems.
  • Instant vs professional. An instant clone does not change the model. A professional clone needs more audio and a fine-tuned model.
  • Detection vs protection. Detection, such as AI Detect, finds a clone in a call. Protection, such as DeFake, changes a recording to make cloning harder.

Questions and answers

How much audio does voice cloning need?

A small quantity. The VALL-E research system (2023) used a 3-second recording. ElevenLabs says that less than two minutes of audio can make an instant clone. For its professional clone, it recommends about 30 minutes. In 2023, a reporter used about five minutes of his speech to go through the voice ID of his bank.

Is voice cloning illegal in the United States?

The FCC ruled in February 2024 that a cloned voice in a call is an "artificial" voice under the TCPA. Thus such calls need the prior express consent of the called party. The ruling does not stop the technology itself.

How do you find a cloned voice on a call?

Frequently, you cannot find it by ear. The FBI says that AI-generated content "is often difficult to identify". Listen to the tone and the words. Then find a number for the person on your own and phone to make sure.

Sources

  1. Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersarXiv (Wang et al.), 5 Jan 2023
  2. Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech SynthesisarXiv (Jia et al.), 12 Jun 2018
  3. Voice cloning (ElevenLabs API concepts)ElevenLabs
  4. How I Broke Into a Bank Account With an AI-Generated VoiceVice (Motherboard, Joseph Cox), 23 Feb 2023
  5. Implications of Artificial Intelligence Technologies on Protecting Consumers from Unwanted Robocalls and Robotexts, Declaratory Ruling (FCC 24-17)Federal Communications Commission, 8 Feb 2024
  6. In the Matter of Steve Kramer, Forfeiture Order (FCC 24-104)Federal Communications Commission, 26 Sep 2024
  7. Consumer Reports calls out slapdash AI voice-cloning safeguardsThe Register, 10 Mar 2025
  8. The FTC Voice Cloning Challenge winnersIEEE Spectrum, 30 May 2024
  9. Senior US Officials Impersonated in Malicious Messaging Campaign (PSA I-051525-PSA)FBI Internet Crime Complaint Center, 15 May 2025

In the news