#Beta - not working currently

n8n Autoscaling System

A security-first, production-ready autoscaling solution for n8n workflow automation. Deploy n8n securely with zero open ports using Cloudflare tunnels, automatic scaling, and enterprise-grade features.

🔒 Why This Setup?

Traditional n8n deployments expose your server directly to the internet. This is dangerous.

This solution provides:

🛡️ Zero Attack Surface: Cloudflare tunnels mean no open ports on your server
🌐 Global DDoS Protection: Cloudflare's network shields your instance
🔐 Automatic HTTPS: Valid SSL certificates without manual configuration
⚡ Auto-scaling: Handle any workload from 1 to hundreds of workflows
🏭 Production Grade: Comprehensive backup/restore, monitoring, and security

🚀 Quick Facts

✅ Battle Tested: Handles hundreds of simultaneous executions on 8-core 16GB VPS
✅ One-Click Deploy: Interactive setup wizard configures everything
✅ Multi-Platform: Auto-detects Docker/Podman, ARM64/AMD64
✅ Enterprise Ready: Backup/restore, monitoring, Tailscale VPN support

This work is an extension of the amazing original setup made by @conor-is-my-name. He detail the further setup here:-

📖 Detailed Guide: https://www.reddit.com/r/n8n/comments/1l9mi6k/major_update_to_n8nautoscaling_build_step_by_step/

When I get around to it I will update the how to in this repo to include more of that information.

Architecture Overview

graph TD
    A[n8n Main] -->|Queues jobs| B[Redis]
    B -->|Monitors queue| C[Autoscaler]
    C -->|Scales| D[n8n Workers]
    B -->|Monitors queue| E[Redis Monitor]
    F[PostgreSQL] -->|Stores data| A
    A -->|Webhooks| G[n8n Webhook]

Features

🔒 Security First

Cloudflare tunnel integration (recommended) - Zero open ports, DDoS protection, automatic HTTPS
Tailscale VPN support - Private network access for teams
Secure password generation - Cryptographically secure secrets with optional salt
Environment isolation - Separate dev/test/production configurations

⚡ Performance & Scaling

Dynamic worker scaling based on Redis queue length
Configurable thresholds for scale-up/down decisions
Multi-architecture support (ARM64/AMD64) with auto-detection
Container runtime flexibility (Docker/Podman) with auto-detection
Performance tuning variables for all components

🛠️ Operations & Maintenance

One-script installation with interactive setup wizard
Comprehensive backup system with smart PostgreSQL backups (full/incremental)
Point-in-time restore with interactive recovery tool
Rclone cloud storage integration (70+ providers)
Health checks for all services
Systemd integration for production deployments

Prerequisites

Required

Docker/Podman and Docker/Podman Compose
Cloudflare account with a domain (free account works fine)
Linux/macOS/WSL2 environment for setup scripts

Recommended Setup Process

Get Docker: For new users, we recommend Docker Desktop or use the Docker convenience script for Ubuntu
Cloudflare Domain: Set up a domain in Cloudflare (can be transferred from another provider)
Cloudflare Tunnel: Create a tunnel token at Cloudflare Zero Trust → Access → Tunnels

🔒 Why Cloudflare Tunnels?

No open ports: Your server stays completely private
Built-in DDoS protection: Cloudflare's global network protects your instance
Free SSL/TLS: Automatic HTTPS with valid certificates
Access control: Optional authentication and access policies
Better performance: Cloudflare's CDN speeds up your workflows

Quick Start (Secure Setup)

Step 1: Cloudflare Tunnel Setup

Create Cloudflare account and add your domain
Navigate to Zero Trust → Access → Tunnels → Create tunnel
Copy your tunnel token (starts with eyJ...)
Configure ingress rules in the tunnel dashboard:
- Add n8n.yourdomain.com → http://n8n:5678
- Add webhook.yourdomain.com → http://n8n-worker:5679
Save configuration - DNS records are created automatically

Step 2: Deploy n8n

Clone this repository:

git clone <repository-url>
cd n8n-autoscaling

Run the interactive setup wizard:
```
./n8n-setup.sh
```
The wizard will guide you through:
- 🔐 Secure password generation
- 🌐 Cloudflare tunnel configuration (recommended)
- 📁 Data directory setup
- 🗄️ Database initialization
- 🔍 Health checks and testing
Start your secure n8n instance:
```
docker compose up -d
```

🎉 That's it! Your n8n instance is now running securely through Cloudflare tunnels.

🏗️ Smart Architecture Selection

The setup wizard automatically chooses the optimal architecture:

🔒 Cloudflare Tunnels (Recommended)

✅ Auto-detected: When tunnel token is configured
✅ No Traefik: Direct tunnel → n8n connection
✅ Zero ports: Maximum security
✅ Command: docker compose -f docker-compose.yml -f docker-compose.cloudflare.yml up -d

🌐 Traditional Setup (Fallback)

⚠️ Used when: No tunnel token configured
⚠️ Includes Traefik: Reverse proxy for port exposure
⚠️ Requires: Firewall configuration and SSL setup
⚠️ Command: docker compose -f docker-compose.yml up -d

Alternative: Direct Access Options

Tailscale VPN: Private network access for teams
Custom reverse proxy: Integration with existing infrastructure

Setup Wizard Features

The n8n-setup.sh script provides:

Interactive Configuration: Step-by-step guided setup
Automatic Path Resolution: Converts relative paths to absolute for Docker compatibility
Environment Management: Create dev/test/production environments
Security: Generates secure random passwords with optional salt
Security Features: Cloudflare tunnel integration (recommended), Tailscale VPN support, secure password generation
Optional Features: Rclone mounts (any cloud storage), external networks, custom reverse proxy
Database Setup: Automatic PostgreSQL and Redis initialization
Health Checks: Verifies services are running correctly
Reset Options: Clean slate functionality if you need to start over
Install systemd Make your installation persistent

Resetting Your Environment

If you need to start fresh or have issues with credentials:

./n8n-setup.sh
# Select option 3: Reset environment

Reset options include:

Everything: Removes all data, .env file, and Docker resources
Just Data: Keeps configuration but removes all database/app data
Just .env: Removes configuration file (warning: existing data won't be accessible)

Rclone Cloud Storage Integration

Optional cloud storage support for data and backups using rclone (70+ providers supported).

Quick Setup

Install rclone: curl https://rclone.org/install.sh | sudo bash
Configure provider: rclone config (setup your Google Drive, S3, etc.)
Create mounts: Setup automatic mounting for data and backups
Enable in n8n: Uncomment rclone variables in .env and use rclone compose override

Supported Providers: Google Drive, OneDrive, Dropbox, AWS S3, Azure Blob, Backblaze B2, SFTP, and many more.

🔒 Security Recommendation

Use rootless containers (Podman/Docker) for best security and simplified permissions with rclone mounts.

📖 Detailed Setup Guide: docs/rclone-mounts.md

Provider-specific examples (Google Drive, S3, OneDrive, SFTP)
Docker vs Podman considerations
Performance tuning for media files
Systemd service configuration
Troubleshooting common issues

Configuration

Make sure you set your own passwords and encryption keys in the .env file if you dont use the setup script
By default each worker handles 10 tasks at a time, you can modify this in the docker-compose under:
- N8N_CONCURRENCY_PRODUCTION_LIMIT=10
Adjust these to be greater than your longest expected workflow execution time measured in seconds:
- N8N_QUEUE_BULL_GRACEFULSHUTDOWNTIMEOUT=300
- N8N_GRACEFUL_SHUTDOWN_TIMEOUT=300

Key Environment Variables

Variable	Description	Default
`MIN_REPLICAS`	Minimum number of worker containers	1
`MAX_REPLICAS`	Maximum number of worker containers	5
`SCALE_UP_QUEUE_THRESHOLD`	Queue length to trigger scale up	5
`SCALE_DOWN_QUEUE_THRESHOLD`	Queue length to trigger scale down	2
`POLLING_INTERVAL_SECONDS`	How often to check queue length	30
`COOLDOWN_PERIOD_SECONDS`	Time between scaling actions	180
`QUEUE_NAME_PREFIX`	Redis queue prefix	`bull`
`QUEUE_NAME`	Redis queue name	`jobs`

n8n Configuration

Ensure these n8n environment variables are set:

EXECUTIONS_MODE=queue
QUEUE_BULL_REDIS_HOST=redis
QUEUE_HEALTH_CHECK_ACTIVE=true

Scaling Behavior

The autoscaler:

Monitors Redis queue length every POLLING_INTERVAL_SECONDS
Scales up when:
- Queue length > SCALE_UP_QUEUE_THRESHOLD
- Current replicas < MAX_REPLICAS
Scales down when:
- Queue length < SCALE_DOWN_QUEUE_THRESHOLD
- Current replicas > MIN_REPLICAS
Respects cooldown period between scaling actions

Monitoring

The system includes:

Redis queue monitor service (redis-monitor)
Docker health checks for all services
Detailed logging from autoscaler

Automatic Updates

For Docker Users

To enable automatic container updates, use Watchtower:

docker run -d \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --schedule "0 0 2 * * *" \
  --cleanup

For Podman Users

Podman auto-update is automatically configured when using the systemd service generator (./generate-systemd.sh). The system will check for updates daily and restart containers with newer images.

Backups

The system includes automated backup functionality with incremental PostgreSQL backups to minimize storage and improve performance.

Backup Components

PostgreSQL: Smart backup system (full every 12h, incremental hourly)
Redis: Database snapshots using BGSAVE (compressed)
n8n Data: Complete data directories including webhook data (compressed)
🔒 Encryption: All backups automatically encrypted using your N8N_ENCRYPTION_KEY

PostgreSQL Backup Strategy

The system uses a sophisticated backup approach:

Full Backups: Complete database dump (larger, standalone restore)
Incremental Backups: WAL (Write-Ahead Log) files only (smaller, faster)
Smart Backup: Automatically chooses full (every 12h) or incremental (hourly)

This approach reduces backup time and storage space while maintaining complete recovery capability.

Running Backups

# Smart backup (recommended - automatically chooses full or incremental)
./backup.sh

# Force specific backup types
./backup.sh postgres-full         # Force full PostgreSQL backup
./backup.sh postgres-incremental  # Force incremental PostgreSQL backup
./backup.sh postgres              # Smart PostgreSQL backup
./backup.sh redis                 # Redis backup only
./backup.sh n8n                   # n8n data backup only

# View help and cron examples
./backup.sh --help

🔒 Backup Encryption

All backups are automatically encrypted using AES-256-CBC encryption with your N8N_ENCRYPTION_KEY. This provides enterprise-grade security for your backup data.

Encryption Features:

Automatic: All backups (PostgreSQL, Redis, n8n data) are encrypted by default
Secure: Uses AES-256-CBC encryption with salt
Key Management: Uses your existing N8N_ENCRYPTION_KEY (same as n8n encryption)
Transparent: Restore script automatically handles encrypted backups

Key Requirements:

Your N8N_ENCRYPTION_KEY must be at least 16 characters
The same key is required for backup and restore operations
Keep your encryption key secure and backed up separately

File Extensions:

Encrypted backups: .gz.enc (e.g., postgres_full_20240115_143022.sql.gz.enc)
Unencrypted backups: .gz (when encryption key not available)

Recommended Cron Schedule

Add to crontab for automated backups:

Option 1: Simple Smart Backups (Recommended)

# Smart backups - full every 12h, incremental hourly
0 * * * * /path/to/n8n-autoscaling/backup.sh >/dev/null 2>&1

Option 2: Separate Service Schedules

# PostgreSQL full backup twice daily
0 0,12 * * * /path/to/n8n-autoscaling/backup.sh postgres-full >/dev/null 2>&1

# PostgreSQL incremental hourly (skip full backup hours)
0 1-11,13-23 * * * /path/to/n8n-autoscaling/backup.sh postgres-incremental >/dev/null 2>&1

# Other services hourly
30 * * * * /path/to/n8n-autoscaling/backup.sh redis >/dev/null 2>&1
45 * * * * /path/to/n8n-autoscaling/backup.sh n8n >/dev/null 2>&1

Setting Up Cron

# Edit your crontab
crontab -e

# Add the recommended line (Option 1)
0 * * * * /path/to/n8n-autoscaling/backup.sh >/dev/null 2>&1

# Save and verify
crontab -l

Rclone Cloud Storage Integration

To enable automatic cloud storage sync:

Uncomment RCLONE_BACKUP_MOUNT in .env
Ensure your rclone remote is mounted at the specified path
Backups will automatically sync to cloud storage and local copies will be removed
7-day retention is maintained on cloud storage

Backup Storage & Sizes

PostgreSQL Full: ~50-500MB (depends on data size)
PostgreSQL Incremental: ~1-50MB (depends on activity)
Redis: ~1-100MB (depends on queue size)
n8n Data: ~10-200MB (depends on workflows and executions)

Storage Locations:

Local: ./backups/{postgres,redis,n8n}/ (if not using rclone cloud storage)
Rclone Cloud Storage: Configured path with automatic cleanup
Retention: 7 days for all backup types

Recovery with Restore Script

The system includes an interactive restore script that automates recovery:

# Interactive restore (recommended)
./restore.sh

# List available backups
./restore.sh --list

# Dry run (see what would be restored)
./restore.sh --dry-run

# Help and options
./restore.sh --help

Restore Script Features:

Interactive Menu: Choose service and backup point
Multi-Source Discovery: Finds backups from both local and rclone cloud storage
🔒 Automatic Decryption: Handles encrypted backups transparently using N8N_ENCRYPTION_KEY
Safety Backup: Creates backup of current data before restore
Integrity Validation: Verifies backup files before restore
Smart Container Management: Safely stops/starts containers
Point-in-Time Recovery: Shows backups with timestamps
Dry Run Mode: Preview restore without making changes

Safety Features:

Multiple confirmation prompts
Automatic safety backup creation
Backup integrity validation
Container state management
Detailed progress reporting

Manual Recovery Examples

# Manual PostgreSQL restore from full backup
gunzip < postgres_full_20240115_143022.sql.gz | docker compose exec -T postgres psql -U postgres -d n8n

# For incremental recovery:
# 1. Restore the latest full backup
# 2. Apply WAL files in chronological order
# (Use restore.sh for automated incremental recovery)

Performance Tuning

The system includes extensive performance tuning options in .env.example. Uncomment and adjust these variables as needed:

n8n Performance

N8N_CONCURRENCY_PRODUCTION_LIMIT: Tasks per worker (default: 10)
N8N_EXECUTIONS_DATA_PRUNE: Enable automatic execution data cleanup
N8N_EXECUTIONS_DATA_MAX_AGE: Keep executions for X hours (default: 336 = 2 weeks)
NODE_OPTIONS: Node.js memory limits and optimization flags
UV_THREADPOOL_SIZE: Node.js thread pool size for I/O operations

PostgreSQL Performance

POSTGRES_SHARED_BUFFERS: Memory for caching data (default: 256MB)
POSTGRES_EFFECTIVE_CACHE_SIZE: Total memory available for caching (default: 1GB)
POSTGRES_WORK_MEM: Memory per query operation (default: 4MB)
POSTGRES_MAX_WORKER_PROCESSES: Background worker processes
Parallel query settings for improved performance on multi-core systems

Redis Performance

REDIS_MAXMEMORY: Maximum memory usage (default: 512mb)
REDIS_MAXMEMORY_POLICY: Eviction policy when memory limit reached
REDIS_SAVE_*: Persistence configuration for snapshots
Connection and networking optimizations

Autoscaler Performance

AUTOSCALER_CPU_LIMIT: CPU limit for autoscaler container
AUTOSCALER_MEMORY_LIMIT: Memory limit for autoscaler container
AUTOSCALER_REDIS_POOL_SIZE: Connection pool size for Redis
AUTOSCALER_DOCKER_TIMEOUT: Timeout for Docker operations

All performance variables are commented out by default. Uncomment and adjust based on your system resources and workload requirements.

Security Best Practices

Recommended: Cloudflare Tunnels (Default)

✅ Zero Attack Surface: No open ports on your server
✅ DDoS Protection: Cloudflare's global network shields your instance
✅ Automatic HTTPS: Valid SSL certificates without configuration
✅ Access Control: Optional authentication and IP restrictions
✅ Audit Logs: Track all access attempts

⚠️ Security Considerations

Container Runtime Security: The autoscaler requires container runtime socket access. Security varies significantly:

Security Ranking (Automatic Detection)

🟢 Rootless Podman - Maximum security, containers run as regular user
🟡 Rootless Docker - Good security with user namespaces
🔴 Rootful Podman - Limited security, some root access
🔴 Rootful Docker - Poor security, full root access equivalent

Automatic Security Features:

Smart Detection: Setup script automatically detects and ranks available container runtimes
Security Warnings: Clear warnings displayed for rootful modes with migration guidance
Migration Instructions: Step-by-step commands to upgrade to more secure configurations
User Confirmation: Explicit acknowledgment required to proceed with less secure setups

Database Security:

PostgreSQL defaults to localhost-only binding for security
Use strong passwords (automatically generated by setup script)
Consider enabling Tailscale for secure remote database access

Backup Security:

✅ Automatic Encryption: All backups encrypted with AES-256-CBC using N8N_ENCRYPTION_KEY
✅ Secure by Default: No configuration needed - encryption happens automatically
⚠️ Key Management: Keep your N8N_ENCRYPTION_KEY secure and backed up separately

🔒 Production Security Checklist

Before deploying to production, ensure:

✅ Authentication & Access

Change all default passwords (setup script enforces this)
Enable Cloudflare tunnels (recommended) or configure firewall rules
Set up Tailscale VPN for team access (optional)
Configure proper user management in n8n

✅ Network Security

PostgreSQL bound to localhost only (default)
Redis password authentication enabled (default)
No unnecessary ports exposed to internet
External network configuration reviewed

✅ Data Protection

Backup encryption configured
SSL/TLS certificates valid
Environment variables secured
Log files protected from unauthorized access

✅ System Hardening

Docker socket access reviewed and understood
Container user permissions verified
System updates applied
Monitoring and alerting configured

Webhook URLs: When using Cloudflare tunnels, webhooks automatically use your secure subdomain:

https://webhook.yourdomain.com/webhook/d7e73b77-6cfb-4add-b454-41e4c91461d8

Security Comparison

Method	Security	Setup Complexity	Public Access	DDoS Protection	Certificate Management
Cloudflare Tunnel	🟢 Excellent	🟢 Easy	✅ Yes	✅ Built-in	✅ Automatic
Tailscale VPN	🟢 Excellent	🟡 Medium	❌ No	❌ None	✅ Automatic
Direct Exposure	🔴 Poor	🔴 Hard	✅ Yes	❌ None	❌ Manual

Recommendation: Use Cloudflare tunnels for production deployments. Only consider alternatives for specific use cases like private team access (Tailscale) or development environments (direct).

Troubleshooting

Cloudflare Tunnel Issues

Tunnel not connecting: Verify your tunnel token is correct and active
DNS not resolving: Check that Cloudflare DNS records were created automatically
502 Bad Gateway: Ensure n8n services are running: docker compose ps
Tunnel status: Check tunnel logs: docker compose logs cloudflared

General Issues

Check container logs: docker compose logs [service]
Verify Redis connection: docker compose exec redis redis-cli -a "${REDIS_PASSWORD}" ping
Check queue length: docker compose exec redis redis-cli -a "${REDIS_PASSWORD}" LLEN bull:jobs:wait
Database connection: docker compose exec postgres pg_isready

Performance Issues

Workers not scaling: Check Redis connection and queue monitoring
Slow responses: Review N8N_CONCURRENCY_PRODUCTION_LIMIT setting
Memory issues: Monitor container resource usage: docker stats

License

MIT License - See LICENSE for details.

License

pie-rs/n8n-autoscaling

About

Resources

License

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Packages