Knowledge Bases
Scrape Websites
Creates a new knowledge base by scraping content from web URLs.
POST
Create a knowledge base by scraping content from one or more web URLs. This is perfect for creating knowledge bases from documentation sites, blog posts, or other web-based content.
Docs for agents: llms.txt
Headers
string
required
Your API key for authentication.
string
required
Must be
application/json for web scraping knowledge bases.Body Parameters
string
default:"web"
required
Must be
"web" for web scraping knowledge bases.string
required
Name for the knowledge base.
string
Optional description of the knowledge base content.
string[]
required
Array of URLs to scrape (maximum 100 URLs).
Response
object
The created knowledge base object.
string
Unique identifier for the knowledge base.
string
Name of the knowledge base.
string
Description of the knowledge base (if provided).
string
Current status:
"PROCESSING", "COMPLETED", "FAILED", or "DELETED".string
Will be
"WEB_SCRAPE" for web scraping knowledge bases.string
Source URLs that were scraped (comma-separated).
string
Base URL derived from the provided URLs.
string
ISO timestamp of creation.
string
ISO timestamp of last update.
string
Error message if status is
"FAILED".null
Will be
null on successful creation.Docs for agents: llms.txt