API Endpoints
Complete reference for ScrapingForge API endpoints - synchronous and asynchronous scraping.
ScrapingForge provides two types of endpoints: synchronous for immediate results and asynchronous for background processing.
Synchronous Scraping
Immediate blocking response with 90-120 second timeout.
POST/GET https://api.scrapingforge.com/api/v1/scraper
Request Methods
- POST: Send parameters in JSON body (recommended for complex requests)
- GET: Send parameters as query strings (simple requests only)
Example Request
curl -X POST https://api.scrapingforge.com/api/v1/scraper \
-H "Authorization: Bearer sf_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"render_js": true,
"premium_proxy": true
}'
Response
<!DOCTYPE html>
<html>
<head>
<title>Example Domain</title>
</head>
<body>
...
</body>
</html>
Asynchronous Scraping
Submit jobs for background processing. Results retained for 30 days.
Submit Job
POST https://api.scrapingforge.com/api/v1/scraper/jobs
curl -X POST https://api.scrapingforge.com/api/v1/scraper/jobs \
-H "Authorization: Bearer sf_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"render_js": true
}'
Response:
{
"job_id": "abc123",
"status": "pending"
}
Check Job Status
GET https://api.scrapingforge.com/api/v1/scraper/jobs/{job_id}
curl https://api.scrapingforge.com/api/v1/scraper/jobs/abc123 \
-H "Authorization: Bearer sf_your_api_key"
Response:
{
"job_id": "abc123",
"status": "completed",
"created_at": "2024-01-01T12:00:00Z",
"completed_at": "2024-01-01T12:00:15Z"
}
Job Statuses
| Status | Description |
|---|---|
pending | Job queued for processing |
running | Job currently being processed |
completed | Job finished successfully |
failed | Job failed (check error message) |
Get Job Result
GET https://api.scrapingforge.com/api/v1/scraper/jobs/{job_id}/result
curl https://api.scrapingforge.com/api/v1/scraper/jobs/abc123/result \
-H "Authorization: Bearer sf_your_api_key"
Workflow Example
# 1. Submit job
JOB_ID=$(curl -X POST https://api.scrapingforge.com/api/v1/scraper/jobs \
-H "Authorization: Bearer sf_your_api_key" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "render_js": true}' \
| jq -r '.job_id')
# 2. Wait and check status
while true; do
STATUS=$(curl https://api.scrapingforge.com/api/v1/scraper/jobs/$JOB_ID \
-H "Authorization: Bearer sf_your_api_key" \
| jq -r '.status')
if [ "$STATUS" = "completed" ]; then
break
fi
sleep 5
done
# 3. Get result
curl https://api.scrapingforge.com/api/v1/scraper/jobs/$JOB_ID/result \
-H "Authorization: Bearer sf_your_api_key"
import requests
import time
api_key = "sf_your_api_key"
base_url = "https://api.scrapingforge.com/api/v1/scraper"
# 1. Submit job
response = requests.post(
f"{base_url}/jobs",
json={"url": "https://example.com", "render_js": True},
headers={"Authorization": f"Bearer {api_key}"}
)
job_id = response.json()["job_id"]
# 2. Poll for completion
while True:
status_response = requests.get(
f"{base_url}/jobs/{job_id}",
headers={"Authorization": f"Bearer {api_key}"}
)
status = status_response.json()["status"]
if status == "completed":
break
time.sleep(5)
# 3. Get result
result = requests.get(
f"{base_url}/jobs/{job_id}/result",
headers={"Authorization": f"Bearer {api_key}"}
)
print(result.text)
const axios = require('axios');
const apiKey = 'sf_your_api_key';
const baseUrl = 'https://api.scrapingforge.com/api/v1/scraper';
async function scrapeAsync() {
// 1. Submit job
const { data: job } = await axios.post(
`${baseUrl}/jobs`,
{ url: 'https://example.com', render_js: true },
{ headers: { Authorization: `Bearer ${apiKey}` } }
);
// 2. Poll for completion
while (true) {
const { data: status } = await axios.get(
`${baseUrl}/jobs/${job.job_id}`,
{ headers: { Authorization: `Bearer ${apiKey}` } }
);
if (status.status === 'completed') break;
await new Promise(resolve => setTimeout(resolve, 5000));
}
// 3. Get result
const { data: result } = await axios.get(
`${baseUrl}/jobs/${job.job_id}/result`,
{ headers: { Authorization: `Bearer ${apiKey}` } }
);
return result;
}