Discourse External Data Plugin

A Discourse plugin that allows you to fetch and store data from external URLs with configurable permissions and refresh schedules.

Features

• Configurable Data Sources: Set up multiple external URLs to fetch data from
• Flexible Scheduling: Configure refresh intervals using simple time formats (e.g., "5 minutes", "1 hour", "2 days")
• Permission Control: Set access levels for each data source (public, authenticated users, moderators, or admins only)
• Request Customization: Add custom headers and query parameters to your requests
• Global Helper: Easy-to-use JavaScript API for theme components to access fetched data
• Admin UI: Comprehensive interface for managing data sources
• Background Jobs: Automatic data fetching with Sidekiq jobs
• Error Handling: Graceful handling of failed requests with error logging

Installation

1. Clone this repository into your Discourse plugins directory:

   cd /var/discourse/plugins
   git clone https://github.com/discourse/discourse-external-data.git

2. Rebuild your Discourse container:

   cd /var/discourse
   ./launcher rebuild app

Configuration

Site Settings

Navigate to Admin > Settings > Plugins and configure:

• external_data_enabled: Enable/disable the plugin
• external_data_max_sources: Maximum number of data sources allowed
• external_data_default_timeout: Default timeout for HTTP requests (in seconds)
• external_data_rate_limit_per_minute: Rate limit for fetch requests
• external_data_max_response_size: Maximum allowed response size in bytes

Admin Interface

1. Go to Admin > Plugins > External Data
2. Click "New Data Source" to create a new source
3. Configure:
   • Name: Friendly name for the data source
   • Key: Unique identifier (lowercase letters, numbers, underscores only)
   • URL: The endpoint to fetch data from
   • Headers: Optional HTTP headers (JSON format)
   • Parameters: Optional query parameters (JSON format)
   • Refresh Schedule: How often to fetch new data
   • Permission Level: Who can access this data
   • Enabled: Toggle to enable/disable the source

Permission Levels

• Public: Anyone can access the data (including anonymous users)
• Authenticated: Only logged-in users can access
• Moderators: Only moderators and admins can access
• Admins: Only administrators can access

Usage in Theme Components

JavaScript API

The plugin provides a global ExternalData object for easy access:

// Basic usage - fetch data by key
ExternalData.get('weather_data').then(data => {
  console.log('Weather data:', data);
}).catch(error => {
  console.error('Failed to fetch weather data:', error);
});

// With options
ExternalData.get('stock_prices', {
  cache: true,      // Use cached data if available (default: true)
  maxAge: 60        // Maximum age of cached data in seconds (default: 300)
}).then(data => {
  // Use the data
});

// Clear cache
ExternalData.clearCache('weather_data'); // Clear specific key
ExternalData.clearCache(); // Clear all cached data

// Check cache status
if (ExternalData.isCached('weather_data')) {
  const cacheInfo = ExternalData.getCacheInfo('weather_data');
  console.log('Cached at:', new Date(cacheInfo.timestamp));
}

In Theme Components

api.modifyClass('component:my-component', {
  async init() {
    this._super(...arguments);
    
    try {
      const data = await ExternalData.get('my_data_source');
      this.set('externalData', data);
    } catch (error) {
      // Handle error
    }
  }
});

In Templates

The data is also accessible via the /external-data/:key endpoint:

// Direct API call
fetch('/external-data/weather_data')
  .then(response => response.json())
  .then(result => {
    console.log('Data:', result.data);
    console.log('Fetched at:', result.fetched_at);
    console.log('Age in seconds:', result.age_seconds);
  });

API Response Format

{
  "key": "weather_data",
  "data": { /* Your fetched data */ },
  "fetched_at": "2024-01-15T10:30:00Z",
  "age_seconds": 120,
  "source": {
    "name": "Weather API",
    "description": "Current weather data"
  }
}

Examples

Example 1: Weather Data

// Configuration
{
  "name": "Weather Data",
  "key": "weather",
  "url": "https://api.openweathermap.org/data/2.5/weather",
  "params": {
    "q": "London",
    "appid": "your-api-key",
    "units": "metric"
  },
  "refresh_schedule": "30 minutes",
  "permission_level": "public"
}

// Usage in theme
ExternalData.get('weather').then(data => {
  const temp = data.main.temp;
  const description = data.weather[0].description;
  // Update UI
});

Example 2: Private API with Authentication

// Configuration
{
  "name": "Company Stats",
  "key": "company_stats",
  "url": "https://api.company.com/stats",
  "headers": {
    "Authorization": "Bearer your-token",
    "Accept": "application/json"
  },
  "refresh_schedule": "1 hour",
  "permission_level": "admins"
}

Development

Running Tests

cd plugins/discourse-external-data
bundle exec rspec

Manual Testing

1. Create a test data source pointing to a reliable API
2. Use the "Fetch Now" button to trigger immediate fetching
3. Check the browser console for the ExternalData helper
4. Monitor Sidekiq jobs in /sidekiq

Troubleshooting

Data Not Updating

1. Check if the data source is enabled
2. Verify the URL is accessible
3. Check Sidekiq logs for job errors
4. Ensure the refresh schedule is valid

Permission Errors

1. Verify the user has the required permission level
2. Check that the user is logged in (for non-public sources)
3. Ensure the data source exists and is enabled

Fetch Failures

1. Check the response in the admin interface
2. Verify headers and parameters are valid JSON
3. Check for timeout issues (increase if needed)
4. Monitor the Rails logs for detailed error messages

Data Explorer Queries

If you have the Data Explorer plugin installed, you can use these SQL queries to analyze your external data:

View All Data Sources and Latest Data

-- View all data sources with their latest successful fetch
SELECT 
    ds.id,
    ds.name,
    ds.key,
    ds.url,
    ds.permission_level,
    ds.enabled,
    ds.refresh_schedule,
    ds.last_fetched_at,
    fd.response_body AS latest_data,
    fd.fetched_at,
    fd.status_code,
    fd.response_time_ms
FROM external_data_sources ds
LEFT JOIN LATERAL (
    SELECT *
    FROM external_fetched_data
    WHERE data_source_id = ds.id
      AND status_code = 200
    ORDER BY fetched_at DESC
    LIMIT 1
) fd ON true
ORDER BY ds.name;

Get Specific Data Source by Key

-- Get data for a specific source
-- :data_source_key (string) - The key of the data source
SELECT 
    ds.name,
    ds.key,
    fd.response_body AS data,
    fd.fetched_at,
    fd.status_code,
    EXTRACT(EPOCH FROM (NOW() - fd.fetched_at)) AS age_seconds
FROM external_data_sources ds
JOIN external_fetched_data fd ON fd.data_source_id = ds.id
WHERE ds.key = :data_source_key
  AND fd.status_code = 200
ORDER BY fd.fetched_at DESC
LIMIT 1;

Data Source Performance Stats

-- Analyze fetch performance and success rates
SELECT 
    ds.name,
    ds.key,
    COUNT(fd.id) AS total_fetches,
    COUNT(CASE WHEN fd.status_code = 200 THEN 1 END) AS successful_fetches,
    ROUND(100.0 * COUNT(CASE WHEN fd.status_code = 200 THEN 1 END) / COUNT(fd.id), 2) AS success_rate,
    AVG(CASE WHEN fd.status_code = 200 THEN fd.response_time_ms END)::INTEGER AS avg_response_time_ms,
    MAX(fd.fetched_at) AS last_fetch,
    ds.enabled
FROM external_data_sources ds
LEFT JOIN external_fetched_data fd ON fd.data_source_id = ds.id
GROUP BY ds.id, ds.name, ds.key, ds.enabled
ORDER BY success_rate DESC, ds.name;

Recent Fetch Errors

-- View recent fetch failures
SELECT 
    ds.name,
    ds.key,
    fd.status_code,
    fd.error_message,
    fd.fetched_at,
    ds.url
FROM external_data_sources ds
JOIN external_fetched_data fd ON fd.data_source_id = ds.id
WHERE fd.status_code != 200
  AND fd.fetched_at > CURRENT_DATE - INTERVAL '7 days'
ORDER BY fd.fetched_at DESC
LIMIT 20;

Data Freshness Report

-- Check how fresh the data is for each source
SELECT 
    ds.name,
    ds.key,
    ds.refresh_schedule,
    ds.last_fetched_at,
    CASE 
        WHEN ds.last_fetched_at IS NULL THEN 'Never fetched'
        WHEN ds.last_fetched_at < NOW() - INTERVAL '1 day' THEN 'Stale (>1 day)'
        WHEN ds.last_fetched_at < NOW() - INTERVAL '1 hour' THEN 'Old (>1 hour)'
        ELSE 'Fresh'
    END AS freshness,
    ds.enabled
FROM external_data_sources ds
ORDER BY 
    CASE 
        WHEN ds.last_fetched_at IS NULL THEN 0
        ELSE EXTRACT(EPOCH FROM (NOW() - ds.last_fetched_at))
    END DESC;

Export Data as JSON

-- Export all active data sources with their latest data as JSON
SELECT json_build_object(
    'sources', json_agg(
        json_build_object(
            'key', ds.key,
            'name', ds.name,
            'data', fd.response_body::json,
            'fetched_at', fd.fetched_at,
            'age_seconds', EXTRACT(EPOCH FROM (NOW() - fd.fetched_at))
        )
    )
) AS export_data
FROM external_data_sources ds
JOIN LATERAL (
    SELECT response_body, fetched_at
    FROM external_fetched_data
    WHERE data_source_id = ds.id
      AND status_code = 200
    ORDER BY fetched_at DESC
    LIMIT 1
) fd ON true
WHERE ds.enabled = true;

Security Considerations

• All URLs are fetched server-side using Discourse's FinalDestination for safety
• Response sizes are limited to prevent memory issues
• Rate limiting prevents abuse
• Permission checks are enforced on every request
• HTTPS is recommended for all external URLs

Contributing

Pull requests are welcome! Please:

1. Follow Discourse coding standards
2. Add tests for new functionality
3. Update documentation as needed
4. Test thoroughly with a real Discourse installation

License

This plugin is licensed under the same license as Discourse.