What Is the Hermes ElevenLabs Voice Agent?
The Hermes ElevenLabs voice agent lets you call your AI assistant on a real phone number and talk to it like a person. You ring a number, it picks up, and it speaks back in a natural voice. It doesn't just chat—it does real work while you talk.
You can be at the gym, in the car, or walking the dog. You call your agent and it writes files, schedules your day, and pulls up past conversations. All without a screen or typing.
In this episode, Julian Goldie demonstrates how the system works. He calls his agent, which uses a voice he picked—an English butler style—and it answers clearly. It feels like talking to a real person, not a machine.
The key insight is that the agent is not just a chatbot. It has tools, memory, and access to your files. When you speak, it listens, thinks, and acts. That is what makes it more than a novelty.
How the Hermes ElevenLabs Voice Agent Works
The system is built from three main pieces that snap together. Each piece has a specific job.
- ElevenLabs is the voice. It makes the agent sound real and reply fast. It handles talking and listening.
- Twilio is the phone number. It lets you actually call your agent. Without it, there is no number to ring.
- Hermes is the brain. It holds your memory, tools, files, and past sessions. It decides what to do with what you say.
You plug the voice and the phone number into Hermes. Now your agent can speak, listen, and act over a normal phone call.
To get started, you need an ElevenLabs key for the voice and a Twilio account for the phone number. Once you have those, the rest falls into place. The team at ElevenLabs even made a full guide for this setup.
Here is a useful trick: you do not have to set it up by hand. You can give the guide to your agent and say, "Hey, set this up for me." It follows the steps for you. You just approve a few permissions, and you are ready.
How to Set Up the Hermes ElevenLabs Voice Agent
Setting up the phone agent is straightforward if you follow the steps. Julian explains the process clearly in the episode.
- Get an ElevenLabs API key for the voice service.
- Get a Twilio account to obtain a phone number.
- Find the official setup guide from the ElevenLabs team.
- Give that guide to your Hermes agent and ask it to set things up for you.
- Approve the permissions it asks for.
- Once done, you can call your number and start talking.
That is the whole process. You do not need to be a coder. If you can hold a phone and talk, you can use this.
Julian also mentions that inside the AI Profit Boardroom, you can get a ready-to-install zip file with Hermes, the phone agent, and talk mode already built in. That skips the manual setup entirely.
Using Your Voice Agent for Real Tasks
Once your agent is set up, you can use it for many everyday tasks. Julian demonstrates several examples in the episode.
- Ask it to write a poem and save it to your downloads folder. It does it and even names the poem.
- Ask it to add a paragraph to an existing file. It finds the file, reads it, and adds new content.
- Ask it to put a Japanese learning session on your calendar for 3 p.m. It sets it up and fixes small errors itself.
- Ask it what you worked on yesterday. It searches your old chats and gives an answer.
- Ask it for a joke. It tells one that actually makes you laugh.
All of this happens over a phone call. No screen, no typing, just your voice.
Another example: you are on your way to the gym and remember new members joined your community. You call and ask your agent to write a welcome message and save it as a text file. It is done before you even park.
The agent is connected to your own setup at home. Even when you are out, you can run your tools, files, and memory like you never left your desk.
Hermes Jarvis: Voice Control Without the Phone
There is a second way to use voice with Hermes, and it does not require a phone call. It is called Hermes Jarvis, and it lives inside the agent operating system.
Instead of calling a number, you click a button and talk. Your agent listens, does the job, and speaks back. Same magic, no phone needed.
You can ask it to open Google, use the browser, or do anything your agent can normally do—just with your voice instead of typing.
One nice touch: every conversation gets saved in your sessions. You can go back later and see what you said and what it did. Nothing gets lost.
For example, at the end of a busy day, you might not remember if you asked your agent to draft coaching call notes. Just scroll through your sessions and check. It is all there.
You can also use it for quick tasks like drafting a social post. Hit the talk button, say "draft me a short post about why AI agents save time and save it to my notes," and it writes and files it while you grab a coffee. One sentence, one task, done.
Speed Tests and Model Choices
Speed is important because the voice has to reply fast or it stops feeling real. The speed depends on the model you pick.
Julian tested several models. Claude 3.5 Haiku was one of the fastest. The replies came back quick and felt smooth. Other models did not perform as well. Qwen did not work well for him, and a Flash model was not fast enough to control the agent the way he wanted.
Here is the simple rule: faster, lighter models give snappy replies. Bigger, heavier models think longer, so the agent feels slower on the call.
Pick your model based on what you are doing. For quick jobs, go light and fast. For complex tasks, you may need a bigger model, but expect a longer wait.
The harder the task, the longer the wait. A simple text file is quick. Building a whole website over the phone will take a while. Match the job to the moment.
Tips, Limits, and Mistakes to Avoid
The best way to use this voice agent is to keep it simple. That is the secret. It shines for small tasks like quick text files, reminders, calendar slots, and pulling up memories.
Do not hand it a giant job over the phone and expect magic. That is not what it is for. It is built for quick actions while you are moving.
One habit that saves time daily: when you are stuck in traffic, call your agent and ask, "Remind me what we covered in last week's coaching call." It searches and tells you. You walk into your day already sharp.
Another limit to keep in mind: the voice model choice matters. If you pick a slow or heavy model, the call will feel clunky. Stick with fast, light models for the best experience.
Also, remember that the agent depends on your setup at home. Make sure your tools and files are accessible. If something is not connected, the agent cannot use it.
Finally, do not be afraid to let the agent set itself up. Giving it the guide and approving permissions is much easier than doing it manually.
Who Is This Voice Agent For?
This voice agent is for anyone who wants to get things done without being tied to a screen. You do not need to be a coder. In the demo, Julian just talked and asked for simple things in plain words. The agent did the rest.
If you can hold a phone and talk, you can use this. It is perfect for busy people who are often on the go—driving, walking, or at the gym—and want to manage tasks hands-free.
It also matters because it changes how you interact with AI. You could always type to an agent, but it did not pick up the phone and talk back live. Now it sits in your pocket like a real assistant you can call.
The voice feels human because you can choose how it sounds. You can make it calm, friendly, or even close to your own voice. That makes it feel less like a stiff machine and more like a real helper.
Julian notes that most other voice services do not work this smoothly with an agent. Hermes works well because it has tool calls, memory, and skills built in—so the voice has something powerful to plug into.
This is moving fast. A year ago, talking to an agent like this felt like science fiction. Now it is a phone call. People who learn it now will be miles ahead.
