​

External API

The External API endpoint allows you to retrieve JSON data from your saved API configurations. This endpoint returns data in your specified format, either from cache or freshly scraped from the source webpage.

​

Endpoints

​

GET Endpoint (Static Data)

GET https://api.pulpminer.com/external/{apiId}

​

POST Endpoint (Dynamic Variables)

POST https://api.pulpminer.com/external/{apiId}

​

Headers

​

Path Parameters

​

Request Body (POST Endpoint Only)

The POST endpoint accepts a JSON object containing dynamic variable values:

{
  "variableName1": "value1",
  "variableName2": "value2"
}

Where:

• Each key corresponds to a variable name you defined in your API configuration
• Each value will replace the corresponding variable in the URL

​

Response

{
  "data": {
    // Your structured JSON data
    "title": "Example Page Title",
    "content": "Main content of the page...",
    "metadata": {
      "author": "John Doe",
      "date": "2024-03-20"
    }
  },
  "errors": null
}

​

Response Fields

​

Error Responses

​

Invalid API Key

{
  "data": null,
  "errors": [
    {
      "message": "invalid key"
    }
  ]
}

​

Invalid API ID

{
  "data": null,
  "errors": [
    {
      "message": "no data"
    }
  ]
}

​

No Credits

{
  "data": null,
  "errors": [
    {
      "message": "No credits left ⚠. Please buy more credits"
    }
  ]
}

​

Server Error

{
  "data": null,
  "errors": [
    {
      "message": "something went wrong"
    }
  ]
}

​

Example Usage

​

Static API (GET)

​

Using cURL

curl -H "apikey: YOUR_API_KEY" https://api.pulpminer.com/external/123

​

Using JavaScript/Node.js

const response = await fetch('https://api.pulpminer.com/external/123', {
  headers: {
    'apikey': 'YOUR_API_KEY'
  }
});

const data = await response.json();

​

Using Python

import requests

headers = {
    'apikey': 'YOUR_API_KEY'
}

response = requests.get(
    'https://api.pulpminer.com/external/123',
    headers=headers
)
data = response.json()

​

Dynamic API with Variables (POST)

​

Using cURL

curl -X POST -H "apikey: YOUR_API_KEY" -H "Content-Type: application/json" \
  -d '{"productId": "12345", "category": "electronics"}' \
  https://api.pulpminer.com/external/123

​

Using JavaScript/Node.js

const response = await fetch('https://api.pulpminer.com/external/123', {
  method: 'POST',
  headers: {
    'apikey': 'YOUR_API_KEY',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    productId: '12345',
    category: 'electronics'
  })
});

const data = await response.json();

​

Using Python

import requests
import json

headers = {
    'apikey': 'YOUR_API_KEY',
    'Content-Type': 'application/json'
}

payload = {
    'productId': '12345',
    'category': 'electronics'
}

response = requests.post(
    'https://api.pulpminer.com/external/123',
    headers=headers,
    data=json.dumps(payload)
)
data = response.json()

​

Dynamic Variables

The dynamic variables feature allows you to create reusable API endpoints with customizable URL parameters. This is especially useful for:

• Fetching data for different products by ID
• Changing search parameters dynamically
• Creating flexible APIs that can target different resources

​

How It Works

1. In the Pulpminer dashboard, enable dynamic variables for your API
2. Define variable names for path segments and query parameters
3. Your dynamic URL will be formatted with variable placeholders like: https://example.com/{{ productId }}/details?category={{ category }}
4. When making a POST request, provide the variable values in the request body
5. The API will substitute your variables into the URL and fetch the data

​

Caching Behavior

The endpoint’s caching behavior depends on your API configuration:

​

Cache Enabled

• Returns cached data if available and less than 15 minutes old
• Automatically refreshes cache in the background if data is old
• Ensures fast response times while maintaining data freshness

​

Cache Disabled

• Always fetches fresh data from the source webpage
• May have slightly longer response times
• Guarantees up-to-date information

​

Notes

• API endpoints use 0.4 credits per request (Without JS Render & Use Session)
• Response format matches your saved API configuration
• Consider enabling caching for better performance
• Monitor your credit usage to avoid interruptions

​

Best Practices

1. Use Appropriate Caching

   • Enable caching for static or slowly changing data
   • Disable caching for real-time data requirements
   • Consider your application’s freshness vs. performance needs

2. Handle Errors

   • Implement proper error handling
   • Use retry logic with exponential backoff
   • Monitor for recurring errors

3. Optimize Usage

   • Cache responses on your end when possible
   • Batch related requests together
   • Implement rate limiting in your application

4. Dynamic Variables Best Practices

   • Use descriptive variable names
   • Document your variable requirements for team members
   • Consider default values for optional parameters

​

Rate Limiting

• 0.4 credits per API request (Without JS Render & Use Session)
• Requests are processed in order
• Consider implementing backoff strategies for retries

​

Monitoring

Monitor your API usage through:

1. Last fetched timestamp in the API dashboard
2. Credit usage tracking
3. Response times and error rates

​

Need Help?

If you encounter issues:

1. Verify your API key and API ID
2. Check your available credits
3. Contact support at hello@pulpminer.com