Interact with the real world mid-call using custom tools.
Custom tools allow your agent to interact with any web API mid-call. Do things like:
Dispatch SMS or emails using the person’s contact info.
Set appointments using live calendar availability.
Generate support tickets in your issue tracker.
Update your CRM with relevant details during the call.
To understand how custom tools work, let’s take a peek under the hood of the Bland AI phone agent.
During the conversation, the phone agent is constantly listening to figure out when it’s supposed to respond. When the phone agent realizes it’s time to respond, it reviews the tools in its toolbox and picks between them.
Those tools include a speak
, wait
, and button press
tool. When you create a custom tool, you add it to the existing ‘toolbox’ for the phone agent to pick from.
A few natural questions arise:
Keep reading to find out.
The next step is to convert the API request into a custom tool. Custom tools have the following properties:
name
- the agent will see this in the list of toolsdescription
- a short explanation of what the tool doesinput_schema
- a JSON schema describing the input dataspeech
(optional) - a string that will be spoken to the agent while your tool waits for a responseresponse_data
- An array of objects that describe how to extract data from the response. Within the response data, you can create variables that the phone agent can reference in its prompt.The agent will see the name in the list of tools. The name, plus the description, help the AI phone agent when it decides which tool to use.
For this example we’ll set the name to BookAppointment
and the description to Books an appointment for the customer
.
The input schema is critical. It defines the shape of the API request, the different inputs the request can take, and also includes an example (which helps our system when creating requests).
Here’s what the input schema could look like:
Two important notes about input schema:
input_schema
is converted into the variable "{{input}}"
that you can use in the request body/query/headers"{{input.property.subproperty}}"
"{{input.service}}"
to have whatever type of appointment that the customer wantsWhat you’re doing here is describing the structure of the variables that the agent will create.
Special Note: If you need to gather detailed letter-by-letter information from the user, raise your interruption_threshold
parameter to about 200
so that the AI doesn’t interject so quickly.
Scroll down to see the full example.
Because requesting external APIs might take a while, we enable you to define a speech
property. The phone agent will say the speech
while it makes the request.
An example speech might look like: Perfect, I'll schedule that right now, give me just a second.
For the restaurant ordering example, the speech could be Thank you, placing that order now.
Once your API request comes back, you need to extract the response data, and then make the phone agent aware of the new information.
The data
field determines how you extract the data while the name
field determines the variable name for reference in the prompt.
Here’s an example response data:
Below is the entire API request for sending a phone call using the outlined custom tool:
How does the phone agent determine what it sends an API request?
The phone agent refers to the input schema and the example
within it. Both of those pieces of information provide context about what data to pass.
Then the phone agent extracts the information from the transcript and passes it to the request body.
You can improve the accuracy of the input data by creating a very clear input_schema
. That includes providing a detailed example
within.
How does the phone agent decide when to use the custom tool?
The phone agent looks at the tool’s name and description. Then it looks at the current context of the conversation to decide whether using the tool makes sense.
How do I access the info the custom tool responds with?
When the API response from the custom tool comes back, you can extract the API response and create variables. You can do this within the response_data
property.
Once you’ve given the variable a name
, you can reference it in the prompt using double brackets ({{}}
).
Note, with the current setup, you might reference variables that have null values. Here’s the restaurant example prompt:
You are taking a drive thru order from a customer. Find out everything that they want like a drive thru cashier. Continue until they say they’re done. Repeat the full order back to them after that, and ask if that’s correct. If they confirm that it’s correct, then and only then will you place their order using the PlaceOrder tool. After you place it, tell them their order total and to pull forward. Their order price is {{order_price}}
In the above prompt, the order_price
will be null until the data comes back. That’s okay though. The prompt is structured to first take the order, then use the PlaceOrder tool, and then finally respond with the order price.
By the time the phone agent is asked for the order price, it will have the information.
Custom tools will continue getting more robust, to further prevent scenarios where variables without value from being referenced in prompts.
If you have any additional questions, reach out at hello@bland.ai and one of our engineers will help.
Interact with the real world mid-call using custom tools.
Custom tools allow your agent to interact with any web API mid-call. Do things like:
Dispatch SMS or emails using the person’s contact info.
Set appointments using live calendar availability.
Generate support tickets in your issue tracker.
Update your CRM with relevant details during the call.
To understand how custom tools work, let’s take a peek under the hood of the Bland AI phone agent.
During the conversation, the phone agent is constantly listening to figure out when it’s supposed to respond. When the phone agent realizes it’s time to respond, it reviews the tools in its toolbox and picks between them.
Those tools include a speak
, wait
, and button press
tool. When you create a custom tool, you add it to the existing ‘toolbox’ for the phone agent to pick from.
A few natural questions arise:
Keep reading to find out.
The next step is to convert the API request into a custom tool. Custom tools have the following properties:
name
- the agent will see this in the list of toolsdescription
- a short explanation of what the tool doesinput_schema
- a JSON schema describing the input dataspeech
(optional) - a string that will be spoken to the agent while your tool waits for a responseresponse_data
- An array of objects that describe how to extract data from the response. Within the response data, you can create variables that the phone agent can reference in its prompt.The agent will see the name in the list of tools. The name, plus the description, help the AI phone agent when it decides which tool to use.
For this example we’ll set the name to BookAppointment
and the description to Books an appointment for the customer
.
The input schema is critical. It defines the shape of the API request, the different inputs the request can take, and also includes an example (which helps our system when creating requests).
Here’s what the input schema could look like:
Two important notes about input schema:
input_schema
is converted into the variable "{{input}}"
that you can use in the request body/query/headers"{{input.property.subproperty}}"
"{{input.service}}"
to have whatever type of appointment that the customer wantsWhat you’re doing here is describing the structure of the variables that the agent will create.
Special Note: If you need to gather detailed letter-by-letter information from the user, raise your interruption_threshold
parameter to about 200
so that the AI doesn’t interject so quickly.
Scroll down to see the full example.
Because requesting external APIs might take a while, we enable you to define a speech
property. The phone agent will say the speech
while it makes the request.
An example speech might look like: Perfect, I'll schedule that right now, give me just a second.
For the restaurant ordering example, the speech could be Thank you, placing that order now.
Once your API request comes back, you need to extract the response data, and then make the phone agent aware of the new information.
The data
field determines how you extract the data while the name
field determines the variable name for reference in the prompt.
Here’s an example response data:
Below is the entire API request for sending a phone call using the outlined custom tool:
How does the phone agent determine what it sends an API request?
The phone agent refers to the input schema and the example
within it. Both of those pieces of information provide context about what data to pass.
Then the phone agent extracts the information from the transcript and passes it to the request body.
You can improve the accuracy of the input data by creating a very clear input_schema
. That includes providing a detailed example
within.
How does the phone agent decide when to use the custom tool?
The phone agent looks at the tool’s name and description. Then it looks at the current context of the conversation to decide whether using the tool makes sense.
How do I access the info the custom tool responds with?
When the API response from the custom tool comes back, you can extract the API response and create variables. You can do this within the response_data
property.
Once you’ve given the variable a name
, you can reference it in the prompt using double brackets ({{}}
).
Note, with the current setup, you might reference variables that have null values. Here’s the restaurant example prompt:
You are taking a drive thru order from a customer. Find out everything that they want like a drive thru cashier. Continue until they say they’re done. Repeat the full order back to them after that, and ask if that’s correct. If they confirm that it’s correct, then and only then will you place their order using the PlaceOrder tool. After you place it, tell them their order total and to pull forward. Their order price is {{order_price}}
In the above prompt, the order_price
will be null until the data comes back. That’s okay though. The prompt is structured to first take the order, then use the PlaceOrder tool, and then finally respond with the order price.
By the time the phone agent is asked for the order price, it will have the information.
Custom tools will continue getting more robust, to further prevent scenarios where variables without value from being referenced in prompts.
If you have any additional questions, reach out at hello@bland.ai and one of our engineers will help.