API Endpoints

Pulpminer API endpoints are unique URLs that return JSON data from your configured webpages. This guide explains how to create, manage, and use your API endpoints effectively.

Creating an Endpoint

Generate JSON
- Enter webpage URL
- Select a scraper (scraper 1, scraper 2, etc.)
- Get AI-generated JSON
- Customize structure if needed
Configure Settings
- Enable/disable caching
- Toggle use session (only available with Scraper 1, establishes a session with the origin URL before connecting to the target URL)
- Configure dynamic variables (optional)
- Select an LLM (AI Model) for JSON generation
- Enter a valid CSS selector to target a section of the webpage
- Provide a Data Extraction Rule for AI
- Review JSON format
- Save configuration
Get Endpoint URL
- Endpoint format: https://api.pulpminer.com/external/{api_id}
- Copy from dashboard
- Note the API ID

Available Scrapers

When creating or editing an API endpoint, you can select from different numbered scrapers. Each scraper is optimized for different types of webpages and content extraction needs. Try different scrapers to find the one that works best for your specific use case.

Managing Endpoints

Dashboard Overview

Your API dashboard shows:

All active endpoints
Source URLs
Cache status
Session state
Dynamic variable status
Last fetch time
Usage statistics

Endpoint Status

Monitor each endpoint’s:

Active/Inactive status
Cache configuration
Session state configuration (Scraper 1 only)
Dynamic variable settings
Last update time
Response format

Using Endpoints

Making Requests

Basic Request (Static)

curl -H "apikey: YOUR_API_KEY" https://api.pulpminer.com/external/123

Dynamic Request (With Variables)

curl -X POST -H "apikey: YOUR_API_KEY" -H "Content-Type: application/json" \
  -d '{"productId": "12345", "category": "electronics"}' \
  https://api.pulpminer.com/external/123

With Error Handling

async function fetchData(apiId, variables = null) {
  try {
    const options = {
      headers: {
        'apikey': 'YOUR_API_KEY'
      }
    };
    
    // If variables are provided, make a POST request
    if (variables) {
      options.method = 'POST';
      options.headers['Content-Type'] = 'application/json';
      options.body = JSON.stringify(variables);
    }
    
    const response = await fetch(
      `https://api.pulpminer.com/external/${apiId}`,
      options
    );
    
    if (!response.ok) {
      const error = await response.json();
      throw new Error(error.errors[0].message);
    }
    
    return await response.json();
  } catch (error) {
    console.error('API Error:', error);
    throw error;
  }
}

// Example usage:
// Static endpoint
fetchData('123');

// Dynamic endpoint with variables
fetchData('123', { productId: '12345', category: 'electronics' });

Dynamic Variables

What Are Dynamic Variables?

Dynamic variables allow you to create flexible API endpoints where parts of the URL path or query parameters can be customized at request time. This is useful for:

Product pages with different IDs
Search results with variable parameters
Category listings with changeable filters
User profiles with different usernames

Setting Up Dynamic Variables

Enable Dynamic Variables
- Toggle the “Enable Dynamic Variables” switch when creating or editing an API
Configure Path Parameters
- Identify segments in the URL path that should be variable
- Toggle the checkbox for each segment you want to make dynamic
- Enter a descriptive variable name for each segment
Configure Query Parameters
- Identify query parameters that should be variable
- Toggle the checkbox for each parameter you want to make dynamic
- Enter a descriptive variable name for each parameter
Preview Dynamic URL
- Review how your dynamic URL will look with variable placeholders
- Format: https://example.com/{{ productId }}/details?category={{ category }}

Using Dynamic Endpoints

To use a dynamic endpoint, send a POST request with a JSON body containing your variable values:

// JavaScript example
const response = await fetch('https://api.pulpminer.com/external/123', {
  method: 'POST',
  headers: {
    'apikey': 'YOUR_API_KEY',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    productId: '12345',
    category: 'electronics'
  })
});

const data = await response.json();

# Python example
import requests
import json

headers = {
    'apikey': 'YOUR_API_KEY',
    'Content-Type': 'application/json'
}

payload = {
    'productId': '12345',
    'category': 'electronics'
}

response = requests.post(
    'https://api.pulpminer.com/external/123',
    headers=headers,
    data=json.dumps(payload)
)
data = response.json()

Response Format

Success Response

{
  "data": {
    "title": "Example Page",
    "content": "Page content...",
    "metadata": {
      "author": "John Doe",
      "date": "2024-03-20"
    }
  },
  "errors": null
}

Error Response

{
  "data": null,
  "errors": [
    {
      "message": "error message here"
    }
  ]
}

Best Practices

1. Endpoint Management

Keep endpoints organized
Monitor usage patterns
Update configurations as needed
Document endpoint purposes

2. Error Handling

Implement proper error handling
Use retry logic
Monitor for failures
Log issues appropriately

3. Performance Optimization

Enable caching when appropriate
Monitor response times
Track usage patterns
Optimize request frequency

4. Dynamic Variable Best Practices

Use descriptive variable names
Document required variables
Consider using default values in your application
Keep variable names consistent across endpoints

Implementation Examples

Node.js Service

class PulpminerService {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.baseUrl = 'https://api.pulpminer.com/external';
  }

  async fetchStaticEndpoint(apiId) {
    try {
      const response = await fetch(`${this.baseUrl}/${apiId}`, {
        headers: {
          'apikey': this.apiKey
        }
      });

      if (!response.ok) {
        const error = await response.json();
        throw new Error(error.errors[0].message);
      }

      return await response.json();
    } catch (error) {
      console.error(`Error fetching endpoint ${apiId}:`, error);
      throw error;
    }
  }
  
  async fetchDynamicEndpoint(apiId, variables) {
    try {
      const response = await fetch(`${this.baseUrl}/${apiId}`, {
        method: 'POST',
        headers: {
          'apikey': this.apiKey,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify(variables)
      });

      if (!response.ok) {
        const error = await response.json();
        throw new Error(error.errors[0].message);
      }

      return await response.json();
    } catch (error) {
      console.error(`Error fetching dynamic endpoint ${apiId}:`, error);
      throw error;
    }
  }
}

Python Client

import requests
from typing import Dict, Any, Optional

class PulpminerClient:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.base_url = 'https://api.pulpminer.com/external'

    def fetch_static_endpoint(self, api_id: str) -> Dict[str, Any]:
        try:
            response = requests.get(
                f'{self.base_url}/{api_id}',
                headers={'apikey': self.api_key}
            )
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            print(f'Error fetching endpoint {api_id}: {e}')
            raise
            
    def fetch_dynamic_endpoint(self, api_id: str, variables: Dict[str, str]) -> Dict[str, Any]:
        try:
            response = requests.post(
                f'{self.base_url}/{api_id}',
                headers={
                    'apikey': self.api_key,
                    'Content-Type': 'application/json'
                },
                json=variables
            )
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            print(f'Error fetching dynamic endpoint {api_id}: {e}')
            raise

Common Use Cases

1. Content Aggregation

// Fetch content from multiple endpoints
async function aggregateContent(endpoints) {
  const results = await Promise.all(
    endpoints.map(id => pulpminer.fetchStaticEndpoint(id))
  );
  return results.map(r => r.data);
}

2. Product Information

// Fetch product details with dynamic variables
async function getProductDetails(productId, category) {
  return await pulpminer.fetchDynamicEndpoint('product-api-id', {
    productId,
    category
  });
}

3. Data Monitoring

# Monitor endpoint for changes
def monitor_endpoint(api_id, interval_seconds):
    while True:
        data = client.fetch_static_endpoint(api_id)
        process_data(data)
        time.sleep(interval_seconds)

4. Data Integration

// Integrate with other systems
async function syncData(apiId, targetSystem) {
  const data = await pulpminer.fetchStaticEndpoint(apiId);
  await targetSystem.update(data);
}

Troubleshooting

Common Issues

Invalid Endpoint
- Verify API ID
- Check endpoint status
- Confirm URL format
- Test with curl
Authentication Errors
- Verify API key
- Check header format
- Confirm key is active
- Test key separately
Data Issues
- Check source webpage
- Verify JSON format
- Review cache settings
- Test endpoint directly
Dynamic Variable Issues
- Verify variable names match those in configuration
- Check Content-Type header is set to application/json
- Ensure all required variables are provided
- Test with curl using the exact variable names

Monitoring

Key Metrics

Response time
Success rate
Error frequency
Usage patterns

Alerts

Set up alerts for:

Endpoint failures
High latency
Error spikes
Usage thresholds

Need Help?

If you’re having endpoint issues:

Check endpoint status
Verify configuration
Test with curl
Contact support at hello@pulpminer.com

Advanced Extraction Options

LLM Selection

When generating JSON, you can choose from multiple Large Language Models (LLMs), such as Gemini 2 (Google) or GPT-4 (OpenAI). The selected LLM determines how the webpage content is interpreted and structured into JSON. Choose the model that best fits your data extraction needs.

CSS Selector Input

You can specify a CSS selector (e.g., ‘body’, ‘.main-content’, ‘#products’) to target a specific section of the webpage for extraction. This allows for more precise data extraction, especially on complex pages. If left blank, the default selector is ‘body’.

Data Extraction Rule for AI

Optionally, provide a Data Extraction Rule to guide the AI/LLM on what information to extract. For example: “Fetch all 20 products”, “Extract all blog post titles”. This helps the AI focus on the most relevant data for your use case.

Tip: Use these advanced options to improve the accuracy and relevance of your API’s JSON output.

Use Session Feature

When ‘Use Session’ is enabled, Pulpminer will first connect to the origin of the target website (for example, for ‘amazon.com/dp/123132’, it will first connect to ‘amazon.com’). It establishes a session (such as cookies or authentication headers) with the origin. Then, using this session, it accesses the full target URL to fetch the required data. This is especially useful for session-protected domains, .gov sites, or websites that require an initial handshake before accessing specific pages. How it works:

User requests data from API with ‘Use Session’ enabled.
Pulpminer connects to the origin URL (e.g., ‘amazon.com’).
A session is established (cookies, headers, etc.).
Using the session, Pulpminer accesses the target URL (e.g., ‘amazon.com/dp/123132’).
Data is fetched from the target URL.
Structured JSON is returned to the user.

Tip: Enable ‘Use Session’ for websites that require session cookies, or are protected by anti-bot measures. This feature is only available with Scraper 1.

Get Started

Core Features

​API Endpoints

​Creating an Endpoint

​Available Scrapers

​Managing Endpoints

​Dashboard Overview

​Endpoint Status

​Using Endpoints

​Making Requests

​Basic Request (Static)

​Dynamic Request (With Variables)

​With Error Handling

​Dynamic Variables

​What Are Dynamic Variables?

​Setting Up Dynamic Variables

​Using Dynamic Endpoints

​Response Format

​Success Response

​Error Response

​Best Practices

​1. Endpoint Management

​2. Error Handling

​3. Performance Optimization

​4. Dynamic Variable Best Practices

​Implementation Examples

​Node.js Service

​Python Client

​Common Use Cases

​1. Content Aggregation

​2. Product Information

​3. Data Monitoring

​4. Data Integration

​Troubleshooting

​Common Issues

​Monitoring

​Key Metrics

​Alerts

​Need Help?

​Advanced Extraction Options

​LLM Selection

​CSS Selector Input

​Data Extraction Rule for AI

​Use Session Feature