feat: improve Fly.io deployment with SQLite storage

- Use SQLite storage instead of PostgreSQL for simpler Fly.io deployment
- Update dagster_fly.yaml with optimized settings for single-container deployment
- Improve deployment scripts with better error handling and validation
- Update documentation with SQLite-specific deployment instructions
- Streamline Docker image for Fly.io environment
- Add support for persistent volume model storage configuration

Author: andrewm4894

README.md

@@ -477,6 +477,7 @@ make fly-deploy               # Deploy with your environment variables
 - 🔐 **Configurable admin credentials**
 - 📊 **Public dashboard** + protected admin interface
 - 🌐 **Global edge deployment** with auto-scaling
+- 🗄️ **SQLite simplicity** - No separate database required!
 
 See [full deployment docs](docs/docs/deployment/fly.md) for detailed instructions.
 

dagster_fly.yaml

@@ -20,12 +20,10 @@ run_launcher:
   module: dagster.core.launcher.default_run_launcher
   class: DefaultRunLauncher
 
-run_storage:
-  module: dagster_postgres.run_storage
-  class: PostgresRunStorage
-  config:
-    postgres_url:
-      env: DATABASE_URL
+# Use SQLite storage instead of PostgreSQL for simpler deployment
+storage:
+  sqlite:
+    base_dir: "/data/dagster_storage"
 
 run_retries:
   enabled: true
@@ -61,22 +59,15 @@ sensors:
   use_threads: true
   num_workers: 2  # Conservative for Fly.io
 
-schedule_storage:
-  module: dagster_postgres.schedule_storage
-  class: PostgresScheduleStorage
-  config:
-    postgres_url:
-      env: DATABASE_URL
-
-event_log_storage:
-  module: dagster_postgres.event_log
-  class: PostgresEventLogStorage
-  config:
-    postgres_url:
-      env: DATABASE_URL
-
 compute_logs:
   module: dagster.core.storage.local_compute_log_manager
   class: LocalComputeLogManager
   config:
-    base_dir: "/tmp/dagster/compute_logs" 
+    base_dir: "/tmp/dagster/compute_logs"
+
+# Local artifact storage for Fly.io persistent volume
+local_artifact_storage:
+  module: dagster.core.storage.root
+  class: LocalArtifactStorage
+  config:
+    base_dir: "/data/artifacts" 

docker/Dockerfile.fly

@@ -34,8 +34,8 @@ RUN ls -la ./dagster_home/ && \
     ls -la /opt/dagster/dagster_home/ && \
     chmod +x /opt/dagster/start.sh
 
-# Create necessary directories
-RUN mkdir -p /data/models /tmp/dagster/compute_logs
+# Create necessary directories for SQLite storage and artifacts
+RUN mkdir -p /data/models /data/dagster_storage /data/artifacts /tmp/dagster/compute_logs
 
 # Environment setup
 ENV DAGSTER_HOME=/opt/dagster/dagster_home

docs/docs/deployment/fly.md

@@ -206,24 +206,23 @@ For more control over the deployment process:
 fly apps create my-anomstack
 ```
 
-### 2. Set Up Database
+### 2. ~~Set Up Database~~ (No longer needed!)
 
-```bash
-# Create PostgreSQL cluster
-fly postgres create --name my-anomstack-db --region ord
+**🎉 SQLite simplification!** Anomstack now uses SQLite storage on the persistent volume, eliminating the need for a separate PostgreSQL database. This makes deployment simpler, faster, and more cost-effective.
 
-# Attach to your app (creates DATABASE_URL secret automatically)
-fly postgres attach my-anomstack-db -a my-anomstack
+```bash
+# No database setup required - SQLite files are created automatically!
+echo "✅ Using SQLite storage on persistent volume"
 ```
 
 ### 3. Create Persistent Volume
 
 ```bash
-# Create 10GB volume for metrics data
+# Create 10GB volume for metrics data and SQLite storage
 fly volumes create anomstack_data --region ord --size 10 -a my-anomstack
 ```
 
-### 4. Configure Secrets (Optional)
+### 3. Configure Secrets (Optional)
 
 Set up alerts and integrations:
 
@@ -247,7 +246,7 @@ fly secrets set \
   -a my-anomstack
 ```
 
-### 5. Deploy
+### 4. Deploy
 
 ```bash
 fly deploy -a my-anomstack

fly.md

@@ -7,7 +7,7 @@
 - **📊 Public Dashboard**: https://anomstack-demo.fly.dev/ (no authentication)
 - **🔐 Protected Dagster**: https://anomstack-demo.fly.dev/dagster (configurable admin credentials)
 - **🌐 nginx Reverse Proxy**: Routes and protects services
-- **🗄️ Managed PostgreSQL**: Fly.io hosted database
+- **🗄️ SQLite Storage**: Simple, reliable file-based storage on persistent volume
 - **📦 Persistent Volume**: 10GB for DuckDB and models
 
 ## Quick Deploy
@@ -53,7 +53,7 @@ fly secrets set ANOMSTACK_ALERT_EMAIL_TO="[email protected]"
 - `fly.toml` - App configuration
 - `nginx.conf` - Reverse proxy with auth
 - `docker/Dockerfile.fly` - Multi-service container
-- `dagster_fly.yaml` - PostgreSQL Dagster config
+- `dagster_fly.yaml` - SQLite Dagster config
 - `scripts/deployment/start.sh` - Service startup script
 
 ## Access

scripts/deployment/deploy_fly.sh

@@ -102,19 +102,8 @@ else
     echo "✅ Volume anomstack_data already exists"
 fi
 
-# Create Postgres database
-echo "🗄️  Setting up PostgreSQL database..."
-if ! fly postgres list | grep -q "$APP_NAME-db"; then
-    echo "Creating new Postgres cluster..."
-    fly postgres create --name "$APP_NAME-db" --region ord --initial-cluster-size 1 --vm-size shared-cpu-1x --volume-size 10
-    echo "✅ PostgreSQL database created"
-else
-    echo "✅ PostgreSQL database already exists"
-fi
-
-# Attach the database to the app
-echo "🔗 Attaching database to app..."
-fly postgres attach "$APP_NAME-db" -a "$APP_NAME" || echo "Database may already be attached"
+# Note: Using SQLite storage on persistent volume instead of PostgreSQL
+echo "📁 SQLite storage will be created automatically on persistent volume at /data/dagster_storage"
 
 echo "⚙️  Setting environment variables..."
 

scripts/deployment/validate_fly_config.sh

@@ -67,6 +67,12 @@ if [[ -f "dagster_fly.yaml" ]]; then
         echo "❌ Not using DefaultRunLauncher - this won't work on Fly.io"
         exit 1
     fi
+    
+    if grep -q "sqlite:" dagster_fly.yaml; then
+        echo "✅ Using SQLite storage (simplified deployment)"
+    else
+        echo "⚠️  Warning: Not using SQLite storage - PostgreSQL dependency required"
+    fi
 else
     echo "❌ dagster_fly.yaml not found"
     exit 1