API Endpoints

Pulpminer API endpoints are unique URLs that return JSON data from your configured webpages. This guide explains how to create, manage, and use your API endpoints effectively.

Creating an Endpoint

Generate JSON
- Enter webpage URL
- Select a scraper (scraper 1, scraper 2, etc.)
- Get AI-generated JSON
- Customize structure if needed
Configure Settings
- Enable/disable caching
- Toggle use session (only available with Scraper 1, establishes a session with the origin URL before connecting to the target URL)
- Configure dynamic variables (optional)
- Review JSON format
- Save configuration
Get Endpoint URL
- Endpoint format: https://api.pulpminer.com/external/{api_id}
- Copy from dashboard
- Note the API ID

Available Scrapers

When creating or editing an API endpoint, you can select from different numbered scrapers. Each scraper is optimized for different types of webpages and content extraction needs. Try different scrapers to find the one that works best for your specific use case.

Managing Endpoints

Dashboard Overview

Your API dashboard shows:

All active endpoints
Source URLs
Cache status
Session state
Dynamic variable status
Last fetch time
Usage statistics

Endpoint Status

Monitor each endpoint’s:

Active/Inactive status
Cache configuration
Session state configuration (Scraper 1 only)
Dynamic variable settings
Last update time
Response format

Using Endpoints

Making Requests

Basic Request (Static)

curl -H "apikey: YOUR_API_KEY" https://api.pulpminer.com/external/123

Dynamic Request (With Variables)

curl -X POST -H "apikey: YOUR_API_KEY" -H "Content-Type: application/json" \
  -d '{"productId": "12345", "category": "electronics"}' \
  https://api.pulpminer.com/external/123

With Error Handling

async function fetchData(apiId, variables = null) {
  try {
    const options = {
      headers: {
        'apikey': 'YOUR_API_KEY'
      }
    };
    
    // If variables are provided, make a POST request
    if (variables) {
      options.method = 'POST';
      options.headers['Content-Type'] = 'application/json';
      options.body = JSON.stringify(variables);
    }
    
    const response = await fetch(
      `https://api.pulpminer.com/external/${apiId}`,
      options
    );
    
    if (!response.ok) {
      const error = await response.json();
      throw new Error(error.errors[0].message);
    }
    
    return await response.json();
  } catch (error) {
    console.error('API Error:', error);
    throw error;
  }
}

// Example usage:
// Static endpoint
fetchData('123');

// Dynamic endpoint with variables
fetchData('123', { productId: '12345', category: 'electronics' });

Dynamic Variables

What Are Dynamic Variables?

Dynamic variables allow you to create flexible API endpoints where parts of the URL path or query parameters can be customized at request time. This is useful for:

Product pages with different IDs
Search results with variable parameters
Category listings with changeable filters
User profiles with different usernames

Setting Up Dynamic Variables

Enable Dynamic Variables
- Toggle the “Enable Dynamic Variables” switch when creating or editing an API
Configure Path Parameters
- Identify segments in the URL path that should be variable
- Toggle the checkbox for each segment you want to make dynamic
- Enter a descriptive variable name for each segment
Configure Query Parameters
- Identify query parameters that should be variable
- Toggle the checkbox for each parameter you want to make dynamic
- Enter a descriptive variable name for each parameter
Preview Dynamic URL
- Review how your dynamic URL will look with variable placeholders
- Format: https://example.com/{{ productId }}/details?category={{ category }}

Using Dynamic Endpoints

To use a dynamic endpoint, send a POST request with a JSON body containing your variable values:

// JavaScript example
const response = await fetch('https://api.pulpminer.com/external/123', {
  method: 'POST',
  headers: {
    'apikey': 'YOUR_API_KEY',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    productId: '12345',
    category: 'electronics'
  })
});

const data = await response.json();

# Python example
import requests
import json

headers = {
    'apikey': 'YOUR_API_KEY',
    'Content-Type': 'application/json'
}

payload = {
    'productId': '12345',
    'category': 'electronics'
}

response = requests.post(
    'https://api.pulpminer.com/external/123',
    headers=headers,
    data=json.dumps(payload)
)
data = response.json()

Response Format

Success Response

{
  "data": {
    "title": "Example Page",
    "content": "Page content...",
    "metadata": {
      "author": "John Doe",
      "date": "2024-03-20"
    }
  },
  "errors": null
}

Error Response

{
  "data": null,
  "errors": [
    {
      "message": "error message here"
    }
  ]
}

Best Practices

1. Endpoint Management

Keep endpoints organized
Monitor usage patterns
Update configurations as needed
Document endpoint purposes

2. Error Handling

Implement proper error handling
Use retry logic
Monitor for failures
Log issues appropriately

3. Performance Optimization

Enable caching when appropriate
Monitor response times
Track usage patterns
Optimize request frequency

4. Dynamic Variable Best Practices

Use descriptive variable names
Document required variables
Consider using default values in your application
Keep variable names consistent across endpoints

Implementation Examples

Node.js Service

class PulpminerService {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.baseUrl = 'https://api.pulpminer.com/external';
  }

  async fetchStaticEndpoint(apiId) {
    try {
      const response = await fetch(`${this.baseUrl}/${apiId}`, {
        headers: {
          'apikey': this.apiKey
        }
      });

      if (!response.ok) {
        const error = await response.json();
        throw new Error(error.errors[0].message);
      }

      return await response.json();
    } catch (error) {
      console.error(`Error fetching endpoint ${apiId}:`, error);
      throw error;
    }
  }
  
  async fetchDynamicEndpoint(apiId, variables) {
    try {
      const response = await fetch(`${this.baseUrl}/${apiId}`, {
        method: 'POST',
        headers: {
          'apikey': this.apiKey,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify(variables)
      });

      if (!response.ok) {
        const error = await response.json();
        throw new Error(error.errors[0].message);
      }

      return await response.json();
    } catch (error) {
      console.error(`Error fetching dynamic endpoint ${apiId}:`, error);
      throw error;
    }
  }
}

Python Client

import requests
from typing import Dict, Any, Optional

class PulpminerClient:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.base_url = 'https://api.pulpminer.com/external'

    def fetch_static_endpoint(self, api_id: str) -> Dict[str, Any]:
        try:
            response = requests.get(
                f'{self.base_url}/{api_id}',
                headers={'apikey': self.api_key}
            )
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            print(f'Error fetching endpoint {api_id}: {e}')
            raise
            
    def fetch_dynamic_endpoint(self, api_id: str, variables: Dict[str, str]) -> Dict[str, Any]:
        try:
            response = requests.post(
                f'{self.base_url}/{api_id}',
                headers={
                    'apikey': self.api_key,
                    'Content-Type': 'application/json'
                },
                json=variables
            )
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            print(f'Error fetching dynamic endpoint {api_id}: {e}')
            raise

Common Use Cases

1. Content Aggregation

// Fetch content from multiple endpoints
async function aggregateContent(endpoints) {
  const results = await Promise.all(
    endpoints.map(id => pulpminer.fetchStaticEndpoint(id))
  );
  return results.map(r => r.data);
}

2. Product Information

// Fetch product details with dynamic variables
async function getProductDetails(productId, category) {
  return await pulpminer.fetchDynamicEndpoint('product-api-id', {
    productId,
    category
  });
}

3. Data Monitoring

# Monitor endpoint for changes
def monitor_endpoint(api_id, interval_seconds):
    while True:
        data = client.fetch_static_endpoint(api_id)
        process_data(data)
        time.sleep(interval_seconds)

4. Data Integration

// Integrate with other systems
async function syncData(apiId, targetSystem) {
  const data = await pulpminer.fetchStaticEndpoint(apiId);
  await targetSystem.update(data);
}

Troubleshooting

Common Issues

Invalid Endpoint
- Verify API ID
- Check endpoint status
- Confirm URL format
- Test with curl
Authentication Errors
- Verify API key
- Check header format
- Confirm key is active
- Test key separately
Data Issues
- Check source webpage
- Verify JSON format
- Review cache settings
- Test endpoint directly
Dynamic Variable Issues
- Verify variable names match those in configuration
- Check Content-Type header is set to application/json
- Ensure all required variables are provided
- Test with curl using the exact variable names

Monitoring

Key Metrics

Response time
Success rate
Error frequency
Usage patterns

Alerts

Set up alerts for:

Endpoint failures
High latency
Error spikes
Usage thresholds

Need Help?

If you’re having endpoint issues:

Check endpoint status
Verify configuration
Test with curl
Contact support at hello@pulpminer.com

Get Started

Core Features

API Endpoints

API Endpoints

Creating an Endpoint

Available Scrapers

Managing Endpoints

Dashboard Overview

Endpoint Status

Using Endpoints

Making Requests

Basic Request (Static)

Dynamic Request (With Variables)

With Error Handling

Dynamic Variables

What Are Dynamic Variables?

Setting Up Dynamic Variables

Using Dynamic Endpoints

Response Format

Success Response

Error Response

Best Practices

1. Endpoint Management

2. Error Handling

3. Performance Optimization

4. Dynamic Variable Best Practices

Implementation Examples

Node.js Service

Python Client

Common Use Cases

1. Content Aggregation

2. Product Information

3. Data Monitoring

4. Data Integration

Troubleshooting

Common Issues

Monitoring

Key Metrics

Alerts

Need Help?

Get Started

Core Features

​API Endpoints

​Creating an Endpoint

​Available Scrapers

​Managing Endpoints

​Dashboard Overview

​Endpoint Status

​Using Endpoints

​Making Requests

​Basic Request (Static)

​Dynamic Request (With Variables)

​With Error Handling

​Dynamic Variables

​What Are Dynamic Variables?

​Setting Up Dynamic Variables

​Using Dynamic Endpoints

​Response Format

​Success Response

​Error Response

​Best Practices

​1. Endpoint Management

​2. Error Handling

​3. Performance Optimization

​4. Dynamic Variable Best Practices

​Implementation Examples

​Node.js Service

​Python Client

​Common Use Cases

​1. Content Aggregation

​2. Product Information

​3. Data Monitoring

​4. Data Integration

​Troubleshooting

​Common Issues

​Monitoring

​Key Metrics

​Alerts

​Need Help?

API Endpoints

Creating an Endpoint

Available Scrapers

Managing Endpoints

Dashboard Overview

Endpoint Status

Using Endpoints

Making Requests

Basic Request (Static)

Dynamic Request (With Variables)

With Error Handling

Dynamic Variables

What Are Dynamic Variables?

Setting Up Dynamic Variables

Using Dynamic Endpoints

Response Format

Success Response

Error Response

Best Practices

1. Endpoint Management

2. Error Handling

3. Performance Optimization

4. Dynamic Variable Best Practices

Implementation Examples

Node.js Service

Python Client

Common Use Cases

1. Content Aggregation

2. Product Information

3. Data Monitoring

4. Data Integration

Troubleshooting

Common Issues

Monitoring

Key Metrics

Alerts

Need Help?