Skip to main content
POST
/
v1
/
knowledge
/
crawl
curl -X POST https://api.bland.ai/v1/knowledge/crawl \
  -H "authorization: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

{
  "data": {
    "urls": [
      "https://example.com/",
      "https://example.com/about",
      "https://example.com/products",
      "https://example.com/contact",
      "https://example.com/docs/getting-started",
      "https://example.com/docs/api-reference",
      "https://example.com/docs/tutorials"
    ],
    "sitemap_url": "https://example.com/sitemap.xml"
  },
  "errors": null
}

Documentation Index

Fetch the complete documentation index at: https://docs.bland.ai/llms.txt

Use this file to discover all available pages before exploring further.

Analyzes a website to discover available URLs from its sitemap. This is useful for finding all the pages available on a website before creating a web scraping knowledge base.

​
Headers

​
authorization
string
required
Your API key for authentication.
​
content-type
string
required
Must be application/json.

​
Body Parameters

​
url
string
required
The base URL of the website to discover sitemap URLs from.

​
Response

​
data
object
Discovered URLs and sitemap information.
​
data.urls
string[]
Array of discovered URLs from the website’s sitemap.
​
data.sitemap_url
string
The sitemap URL that was found and processed (if any).
​
errors
null
Will be null on successful discovery.
curl -X POST https://api.bland.ai/v1/knowledge/crawl \
  -H "authorization: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

{
  "data": {
    "urls": [
      "https://example.com/",
      "https://example.com/about",
      "https://example.com/products",
      "https://example.com/contact",
      "https://example.com/docs/getting-started",
      "https://example.com/docs/api-reference",
      "https://example.com/docs/tutorials"
    ],
    "sitemap_url": "https://example.com/sitemap.xml"
  },
  "errors": null
}
Docs for agents: llms.txt