Simplify Docker architecture and remove Docker Hub dependencies

- Remove all andrewm4894/anomstack image references, force local builds
- Switch from DockerRunLauncher to DefaultRunLauncher for simpler deployment
- Replace PostgreSQL with SQLite storage for reduced complexity
- Remove Docker socket and complex volume mounts
- Update Makefile to remove Docker Hub commands (tag, push, pull)
- Simplify docker-smart command to build locally instead of pulling
- Update DOCKER.md to reflect simplified 4-container architecture
- Align Docker setup with Fly.io approach for consistency

Benefits:
- Fewer resources required (no PostgreSQL container)
- Simpler troubleshooting (fewer moving parts)
- Users always get latest code (local builds)
- Easier deployment and development setup

Author: andrewm4894

DOCKER.md

@@ -4,15 +4,21 @@ This document provides comprehensive information about the Docker setup for the
 
 ## Overview
 
-Anomstack uses Docker to containerize all components for easy deployment and development. The setup includes:
+Anomstack uses a **simplified Docker architecture** with local image building and SQLite storage for easy deployment and development. The setup includes:
 
 - **Dagster Code Server**: Runs your data pipelines
-- **Dagster UI (Dagit)**: Web interface for pipeline management
-- **Dagster Daemon**: Background process for scheduling and execution
+- **Dagster UI**: Web interface for pipeline management  
+- **Dagster Daemon**: Background process for scheduling and execution (uses DefaultRunLauncher)
 - **Dashboard**: FastHTML-based metrics dashboard
-- **PostgreSQL**: Database for Dagster metadata
+- **SQLite Storage**: All Dagster metadata stored in SQLite files (no separate database)
 - **DuckDB Volume**: Persistent storage for metrics data
 
+### Key Simplifications
+- ✅ **No PostgreSQL**: Uses SQLite for simpler deployment
+- ✅ **No Docker Socket**: DefaultRunLauncher runs jobs in same container
+- ✅ **Local Builds**: All images built locally (no Docker Hub dependency)
+- ✅ **Fewer Resources**: Reduced memory and complexity
+
 ## Quick Start
 
 ```bash
@@ -27,39 +33,32 @@ make docker
 
 ## Services
 
-### 1. PostgreSQL Database (`anomstack_postgresql`)
-- **Image**: `postgres:11`
-- **Purpose**: Stores Dagster run history, schedules, and metadata
-- **Port**: 5432 (configurable via `ANOMSTACK_POSTGRES_FORWARD_PORT`)
-- **Environment Variables**:
-  - `ANOMSTACK_POSTGRES_USER` (default: postgres_user)
-  - `ANOMSTACK_POSTGRES_PASSWORD` (default: postgres_password)
-  - `ANOMSTACK_POSTGRES_DB` (default: postgres_db)
-
-### 2. Code Server (`anomstack_code`)
-- **Image**: `andrewm4894/anomstack_code:latest`
+### 1. Code Server (`anomstack_code`)
+- **Image**: Built locally from `docker/Dockerfile.anomstack_code`
 - **Purpose**: Runs Dagster user code via gRPC
 - **Port**: 4000 (internal)
 - **Restart Policy**: `always`
 - **Volumes**:
   - `./tmp:/opt/dagster/app/tmp` (temporary files)
   - `anomstack_metrics_duckdb:/data` (DuckDB data)
+  - `./dagster_home:/opt/dagster/dagster_home` (Dagster storage - includes SQLite)
 
-### 3. Dagster UI (`anomstack_dagit`)
-- **Image**: `andrewm4894/anomstack_dagster:latest`
-- **Purpose**: Web interface for pipeline management
+### 2. Dagster UI (`anomstack_webserver`)
+- **Image**: Built locally from `docker/Dockerfile.dagster`
+- **Purpose**: Web interface for pipeline management  
 - **Port**: 3000 (external)
 - **Restart Policy**: `on-failure`
 - **Access**: http://localhost:3000
+- **Storage**: Uses SQLite files in mounted dagster_home volume
 
-### 4. Dagster Daemon (`anomstack_daemon`)
-- **Image**: `andrewm4894/anomstack_dagster:latest`
+### 3. Dagster Daemon (`anomstack_daemon`) 
+- **Image**: Built locally from `docker/Dockerfile.dagster`
 - **Purpose**: Background process for scheduling and run execution
 - **Restart Policy**: `on-failure`
-- **Volumes**: Same as dagit (for Docker socket access)
+- **Run Launcher**: DefaultRunLauncher (runs jobs in same container process)
 
-### 5. Dashboard (`anomstack_dashboard`)
-- **Image**: `andrewm4894/anomstack_dashboard:latest`
+### 4. Dashboard (`anomstack_dashboard`)
+- **Image**: Built locally from `docker/Dockerfile.anomstack_dashboard`
 - **Purpose**: FastHTML-based metrics dashboard
 - **Port**: 5001 (external) → 8080 (internal)
 - **Restart Policy**: `on-failure`
@@ -76,8 +75,13 @@ make docker
 
 ### Bind Mounts
 - **`./tmp`**: Temporary files and local artifacts
-- **`/var/run/docker.sock`**: Docker socket for container management
-- **`/tmp/io_manager_storage`**: Dagster I/O manager storage
+- **`./dagster_home`**: Dagster configuration, SQLite storage, and logs  
+- **`./metrics`**: Hot-reloadable metric configurations
+
+### Storage Simplification
+- **SQLite Storage**: All Dagster metadata (runs, schedules, events) stored in SQLite files
+- **No PostgreSQL**: Eliminated separate database container for simpler deployment
+- **No Docker Socket**: DefaultRunLauncher removes need for Docker socket access
 
 ## Environment Configuration
 
@@ -87,15 +91,8 @@ Create a `.env` file based on `.example.env`:
 ```bash
 # Database paths
 ANOMSTACK_DUCKDB_PATH=/data/anomstack.db
-ANOMSTACK_SQLITE_PATH=tmpdata/anomstack-sqlite.db
-
-# PostgreSQL
-ANOMSTACK_POSTGRES_USER=postgres_user
-ANOMSTACK_POSTGRES_PASSWORD=postgres_password
-ANOMSTACK_POSTGRES_DB=postgres_db
-ANOMSTACK_POSTGRES_FORWARD_PORT=5432
 
-# Dagster
+# Dagster (uses SQLite for metadata storage)
 DAGSTER_HOME=
 DAGSTER_LOG_LEVEL=DEBUG
 DAGSTER_CONCURRENCY=4
@@ -117,10 +114,7 @@ make docker-prune             # Remove everything including volumes (⚠️ dele
 ### Image Management
 ```bash
 make docker-build             # Build all images locally
-make docker-tag               # Tag images for Docker Hub
-make docker-push              # Push images to Docker Hub
-make docker-build-push        # Build, tag, and push in one command
-make docker-pull              # Pull latest images from Docker Hub
+make docker-smart             # Build and start services (recommended)
 ```
 
 ### Debugging
@@ -300,17 +294,21 @@ make docker-clean
 - Set up log aggregation
 - Monitor volume usage
 
-## Docker Hub Images
+## Building Images
 
-All images are available on Docker Hub:
-- `andrewm4894/anomstack_code:latest`
-- `andrewm4894/anomstack_dagster:latest`
-- `andrewm4894/anomstack_dashboard:latest`
+All images are built locally from their respective Dockerfiles:
+- `anomstack_code_image` - Built from `docker/Dockerfile.anomstack_code`
+- `anomstack_dagster_image` - Built from `docker/Dockerfile.dagster`  
+- `anomstack_dashboard_image` - Built from `docker/Dockerfile.anomstack_dashboard`
 
-### Building and Pushing
-Images are automatically tagged and pushed using the Makefile:
+### Building Images
+Images are built automatically when using Docker Compose:
 ```bash
-make docker-build-push
+# Build and start all services
+make docker-smart
+
+# Or build manually
+docker compose build --no-cache
 ```
 
-This ensures consistent deployments across environments.
+This ensures you're always running the latest code and dependencies.

Makefile

@@ -30,25 +30,20 @@ dev:
 # DOCKER OPERATIONS
 # =============================================================================
 
-.PHONY: docker docker-dev docker-smart docker-build docker-dev-build docker-tag docker-push docker-build-push
-.PHONY: docker-pull docker-clean docker-logs docker-logs-code docker-logs-dagit docker-logs-daemon docker-logs-dashboard
+.PHONY: docker docker-dev docker-smart docker-build docker-dev-build docker-clean 
+.PHONY: docker-logs docker-logs-code docker-logs-dagit docker-logs-daemon docker-logs-dashboard
 .PHONY: docker-shell-code docker-shell-dagit docker-shell-dashboard docker-restart-dashboard docker-restart-code docker-restart reload-config enable-auto-reload enable-config-watcher
 .PHONY: docker-stop docker-down docker-rm docker-prune
 
 # start docker containers (now uses pre-built images)
 docker:
 	docker compose up -d
 
-# smart docker start: try to pull, fallback to build if images don't exist
+# smart docker start: build images locally and start containers
 docker-smart:
-	@echo "🔄 Attempting to pull pre-built images..."
-	@if docker compose pull 2>/dev/null; then \
-		echo "✅ Successfully pulled images, starting containers..."; \
-		docker compose up -d; \
-	else \
-		echo "⚠️  Pull failed, building images locally..."; \
-		make docker-dev-build && make docker-dev; \
-	fi
+	@echo "🔄 Building images locally and starting containers..."
+	docker compose build --no-cache
+	docker compose up -d
 
 # start docker containers with local development images
 docker-dev:
@@ -64,26 +59,29 @@ docker-build:
 docker-dev-build:
 	docker compose -f docker-compose.yaml -f docker-compose.dev.yaml build --no-cache
 
+# The following commands are commented out as we no longer publish to Docker Hub
+# Users should build images locally instead
+
 # tag docker images for Docker Hub
-docker-tag:
-	docker tag anomstack_code_image andrewm4894/anomstack_code:latest
-	docker tag anomstack_dagster_image andrewm4894/anomstack_dagster:latest
-	docker tag anomstack_dashboard_image andrewm4894/anomstack_dashboard:latest
+# docker-tag:
+# 	docker tag anomstack_code_image andrewm4894/anomstack_code:latest
+# 	docker tag anomstack_dagster_image andrewm4894/anomstack_dagster:latest
+# 	docker tag anomstack_dashboard_image andrewm4894/anomstack_dashboard:latest
 
 # push docker images to Docker Hub
-docker-push:
-	docker push andrewm4894/anomstack_code:latest
-	docker push andrewm4894/anomstack_dagster:latest
-	docker push andrewm4894/anomstack_dashboard:latest
+# docker-push:
+# 	docker push andrewm4894/anomstack_code:latest
+# 	docker push andrewm4894/anomstack_dagster:latest
+# 	docker push andrewm4894/anomstack_dashboard:latest
 
 # build, tag, and push all images in one command
-docker-build-push: docker-build docker-tag docker-push
+# docker-build-push: docker-build docker-tag docker-push
 
 # pull latest images from Docker Hub
-docker-pull:
-	docker pull andrewm4894/anomstack_code:latest
-	docker pull andrewm4894/anomstack_dagster:latest
-	docker pull andrewm4894/anomstack_dashboard:latest
+# docker-pull:
+# 	docker pull andrewm4894/anomstack_code:latest
+# 	docker pull andrewm4894/anomstack_dagster:latest
+# 	docker pull andrewm4894/anomstack_dashboard:latest
 
 # clean up unused docker resources
 docker-clean:

dagster_docker.yaml

@@ -6,58 +6,30 @@ run_coordinator:
   module: dagster.core.run_coordinator
   class: QueuedRunCoordinator
   config:
-    max_concurrent_runs: 10  # Reduced from 25 to prevent excessive storage
+    max_concurrent_runs: 8  # Reduced for better stability in single container
     tag_concurrency_limits:
       - key: "dagster/concurrency_key"
         value: "database"
-        limit: 2  # Reduced from 3
+        limit: 1  # Reduced to prevent resource contention
       - key: "dagster/concurrency_key"
         value: "ml_training"
-        limit: 1  # Reduced from 2
+        limit: 1
 
+# Use DefaultRunLauncher instead of DockerRunLauncher for simpler Docker setup
 run_launcher:
-  module: dagster_docker
-  class: DockerRunLauncher
-  config:
-    env_vars:
-      - DAGSTER_POSTGRES_USER
-      - DAGSTER_POSTGRES_PASSWORD
-      - DAGSTER_POSTGRES_DB
-      - ANOMSTACK_DUCKDB_PATH
-      - ANOMSTACK_TABLE_KEY
-      - ANOMSTACK_MODEL_PATH
-      - ANOMSTACK_IGNORE_EXAMPLES
-      - DAGSTER_HOME
-      - ANOMSTACK_HOME
-    network: anomstack_network
-    container_kwargs:
-      volumes: # Make docker client accessible to any launched containers as well
-        - /var/run/docker.sock:/var/run/docker.sock
-        - /tmp/io_manager_storage:/tmp/io_manager_storage
-        - ${ANOMSTACK_HOME}/tmp:/opt/dagster/app/tmp
-        - ${ANOMSTACK_HOME}/dagster_home:/opt/dagster/dagster_home
-        - ${ANOMSTACK_HOME}/metrics:/opt/dagster/app/metrics
-        - anomstack_metrics_duckdb:/data
+  module: dagster.core.launcher.default_run_launcher
+  class: DefaultRunLauncher
 
-run_storage:
-  module: dagster_postgres.run_storage
-  class: PostgresRunStorage
-  config:
-    postgres_db:
-      hostname: anomstack_postgresql
-      username:
-        env: DAGSTER_POSTGRES_USER
-      password:
-        env: DAGSTER_POSTGRES_PASSWORD
-      db_name:
-        env: DAGSTER_POSTGRES_DB
-      port: 5432
+# Use SQLite storage instead of PostgreSQL for simpler deployment
+storage:
+  sqlite:
+    base_dir: "/opt/dagster/dagster_home/storage"
 
 run_retries:
   enabled: true
-  max_retries: 1
+  max_retries: 2  # Increased for better reliability
 
-# Aggressive retention policies to prevent disk space issues
+# Retention policies to prevent disk space issues
 retention:
   schedule:
     purge_after_days: 3  # Keep schedule runs for 3 days
@@ -73,45 +45,34 @@ run_monitoring:
   start_timeout_seconds: 300   # 5 minutes to start
   cancel_timeout_seconds: 180  # 3 minutes to cancel
   max_runtime_seconds: 900     # 15 minutes max runtime per run
-  max_resume_run_attempts: 2   # Resume runs after worker crashes (DockerRunLauncher only)
   poll_interval_seconds: 60    # Check every minute
 
 # Disable telemetry to reduce disk writes
 telemetry:
   enabled: false
 
+# Optimized for container environment
 schedules:
   use_threads: true
-  num_workers: 8
+  num_workers: 4  # Reasonable for Docker container
 
 sensors:
-  use_threads: true
-  num_workers: 4
+  use_threads: true 
+  num_workers: 2  # Reasonable for Docker container
 
-schedule_storage:
-  module: dagster_postgres.schedule_storage
-  class: PostgresScheduleStorage
+compute_logs:
+  module: dagster.core.storage.local_compute_log_manager
+  class: LocalComputeLogManager
   config:
-    postgres_db:
-      hostname: anomstack_postgresql
-      username:
-        env: DAGSTER_POSTGRES_USER
-      password:
-        env: DAGSTER_POSTGRES_PASSWORD
-      db_name:
-        env: DAGSTER_POSTGRES_DB
-      port: 5432
+    base_dir: "/opt/dagster/dagster_home/compute_logs"
 
-event_log_storage:
-  module: dagster_postgres.event_log
-  class: PostgresEventLogStorage
+# Local artifact storage
+local_artifact_storage:
+  module: dagster.core.storage.root
+  class: LocalArtifactStorage
   config:
-    postgres_db:
-      hostname: anomstack_postgresql
-      username:
-        env: DAGSTER_POSTGRES_USER
-      password:
-        env: DAGSTER_POSTGRES_PASSWORD
-      db_name:
-        env: DAGSTER_POSTGRES_DB
-      port: 5432
+    base_dir: "/opt/dagster/dagster_home/artifacts"
+
+# Enhanced logging for debugging
+code_servers:
+  reload_timeout: 60  # Give code servers more time to reload