Merge pull request #200 from andrewm4894/dont-use-grpc-by-default

Dont use grpc by default

Author: andrewm4894

ARCHITECTURE.md

@@ -13,6 +13,7 @@ Anomstack is a distributed anomaly detection platform built on modern data stack
 - **Flexibility**: Support for multiple data sources, ML algorithms, and deployment patterns
 - **Observability**: Built-in monitoring, logging, and visualization
 - **Simplicity**: Minimal configuration required to get started
+- **Reliability**: Direct Python module loading eliminates gRPC network overhead and improves system stability
 
 ## High-Level Architecture
 
@@ -379,29 +380,34 @@ graph LR
 
 ### Docker Containerized
 
+Simplified 3-container architecture with direct Python module loading (no gRPC server):
+
 ```mermaid
 graph TB
     subgraph "Docker Environment"
-        subgraph "Dagster Container"
-            DG[Dagster]
+        subgraph "Consolidated Container"
+            DGW[Dagster Webserver + User Code<br/>(Direct Python Module Loading)]
         end
 
-        subgraph "Dashboard Container"
-            DASH[FastHTML Dashboard]
+        subgraph "Daemon Container"
+            DGD[Dagster Daemon]
         end
 
-        subgraph "Database Container"
-            PG[(PostgreSQL)]
+        subgraph "Dashboard Container"
+            DASH[FastHTML Dashboard]
         end
 
         subgraph "Storage"
-            VOL[Docker Volumes]
+            SQLITE[(SQLite Files)]
+            DUCKDB[(DuckDB Volume)]
         end
     end
 
-    DG --> PG
-    DG --> VOL
-    DASH --> DG
+    DGW --> SQLITE
+    DGW --> DUCKDB
+    DGD --> SQLITE
+    DGD --> DUCKDB
+    DASH --> DUCKDB
 ```
 
 ### Cloud Native (Dagster Cloud)
@@ -434,6 +440,61 @@ graph TB
     JOBS --> EMAIL
 ```
 
+### Advanced: gRPC Code Server (Optional)
+
+By default, Anomstack uses direct Python module loading for simplicity and reliability. However, users can optionally configure separate gRPC code servers for advanced deployment scenarios such as:
+
+- **Multi-language environments**: When integrating with non-Python services
+- **Container isolation**: When user code needs strict separation from Dagster services  
+- **High-security environments**: When code execution requires additional sandboxing
+- **Distributed deployments**: When code servers need to run on separate machines
+
+#### Enabling gRPC Setup
+
+To enable gRPC code servers, modify your `workspace.yaml`:
+
+```yaml
+load_from:
+  - grpc_server:
+      host: localhost
+      port: 4000
+      location_name: "anomstack_code"
+```
+
+And update your deployment to include a separate code server:
+
+```yaml
+# docker-compose.yaml example
+services:
+  code_server:
+    image: anomstack_code:latest
+    command: dagster code-server start -h 0.0.0.0 -p 4000 -f anomstack/main.py
+    ports:
+      - "4000:4000"
+  
+  dagster_webserver:
+    image: anomstack_dagster:latest
+    depends_on:
+      - code_server
+    ports:
+      - "3000:3000"
+```
+
+#### Trade-offs
+
+**Benefits of gRPC approach:**
+- Stronger isolation between user code and Dagster services
+- Support for non-Python user code
+- Better resource allocation control
+
+**Benefits of direct module loading (default):**
+- Simpler deployment and configuration
+- Eliminates network overhead and potential connectivity issues
+- Faster startup times and more reliable execution
+- Easier debugging and development workflow
+
+For most use cases, the default direct Python module loading is recommended.
+
 ## Security Architecture
 
 ### Authentication & Authorization
@@ -553,14 +614,14 @@ Anomstack provides APIs for external integration:
 
 | Layer | Technology | Purpose |
 |-------|------------|---------|
-| Orchestration | Dagster | Pipeline orchestration and scheduling |
+| Orchestration | Dagster | Pipeline orchestration and scheduling (consolidated architecture) |
 | Web Framework | FastHTML | Dashboard and API development |
 | ML Library | PyOD | Anomaly detection algorithms |
 | Data Processing | Pandas, NumPy | Data manipulation and analysis |
 | Visualization | Plotly, Matplotlib | Chart generation and visualization |
-| Database | PostgreSQL, SQLite | Metadata and configuration storage |
+| Database | SQLite, DuckDB | Metadata and metrics storage (simplified deployment) |
 | Cloud Storage | S3, GCS | Model and data storage |
-| Containerization | Docker | Application packaging and deployment |
-| Monitoring | Prometheus, Grafana | System monitoring and alerting |
+| Containerization | Docker | Application packaging and deployment (3-container architecture) |
+| Code Loading | Direct Python Modules | User code integration (no gRPC server needed) |
 
 This architecture enables Anomstack to provide a robust, scalable, and maintainable anomaly detection platform that can adapt to various deployment scenarios and organizational requirements.

CLAUDE.md

@@ -13,13 +13,18 @@ Anomstack is an open-source anomaly detection system built on Dagster and FastHT
 - `make dashboard` - Start FastHTML dashboard locally (port 5003)
 - `make dashboard-uvicorn` - Start dashboard with uvicorn (hot reload)
 - `make dashboard-local-dev` - Start dashboard with seeded test data
+- `make dev` - Setup local development environment and install dependencies
 
 ### Docker Operations
 - `make docker` - Start all services with Docker Compose
-- `make docker-dev` - Start with local development images
+- `make docker-dev` - Start with local development images  
 - `make docker-smart` - Build fresh images and start containers
+- `make docker-build` - Build all images locally
 - `make docker-logs` - View logs for all containers
+- `make docker-logs-<service>` - View logs for specific service (code, dagit, dashboard, daemon)
+- `make docker-shell-<service>` - Get shell access to running containers
 - `make docker-restart` - Restart all containers (useful for .env changes)
+- `make docker-stop` - Stop all containers
 
 ### Testing & Quality
 - `pytest` or `make tests` - Run test suite
@@ -31,6 +36,17 @@ Anomstack is an open-source anomaly detection system built on Dagster and FastHT
 - `make seed-local-db-all` - Seed with all example metric batches
 - `make seed-local-db-custom BATCHES='batch1,batch2' DB_PATH='path/to/db'`
 
+### Configuration & Hot Reload
+- `make reload-config` - Reload configuration without restarting containers
+- `make enable-auto-reload` - Enable automatic config reloading
+- `make enable-config-watcher` - Enable smart config file watcher
+
+### Reset & Cleanup Operations
+- `make reset-interactive` - Interactive reset with guided options
+- `make reset-gentle` - Rebuild containers (safest reset)
+- `make reset-nuclear` - Remove everything including data
+- `make dagster-cleanup-standard` - Clean up old Dagster runs
+
 ## Architecture
 
 ### Core Components
@@ -99,14 +115,19 @@ ANOMSTACK__PYTHON_INGEST_SIMPLE__ALERT_METHODS=email
 
 ## Development Notes
 
+### Important Development Rule
+**ALWAYS check the Makefile first** when working on Anomstack! The project includes comprehensive Make commands for all common development tasks. Before running manual commands, review the available Makefile commands.
+
 ### Code Style
 - Uses ruff for linting (line length: 100)
-- Star imports allowed in dashboard modules
+- Star imports allowed in dashboard modules (F403/F405 rules ignored)
 - Pre-commit hooks enforce code quality
+- Per-file lint ignores configured for dashboard routes and maintenance scripts
 
 ### Testing
 - Tests in `tests/` directory
-- Use pytest for running tests
+- Use `pytest` or `make tests` for running tests
+- `make test-examples` - Run only example ingest function tests
 - Test coverage tracking with badges in README
 
 ### Deployment Options

DOCKER.md

@@ -4,20 +4,20 @@ This document provides comprehensive information about the Docker setup for the
 
 ## Overview
 
-Anomstack uses a **simplified Docker architecture** with local image building and SQLite storage for easy deployment and development. The setup includes:
+Anomstack uses a **simplified 3-container Docker architecture** with direct Python module loading and SQLite storage for improved reliability and easy deployment. The setup includes:
 
-- **Dagster Code Server**: Runs your data pipelines
-- **Dagster UI**: Web interface for pipeline management  
+- **Dagster Webserver + User Code** (consolidated): Web interface and data pipelines in a single container using direct Python module loading
 - **Dagster Daemon**: Background process for scheduling and execution (uses DefaultRunLauncher)
 - **Dashboard**: FastHTML-based metrics dashboard
 - **SQLite Storage**: All Dagster metadata stored in SQLite files (no separate database)
 - **DuckDB Volume**: Persistent storage for metrics data
 
 ### Key Simplifications
+- ✅ **No gRPC Code Server**: User code loaded directly as Python module (eliminates network overhead and reliability issues)
 - ✅ **No PostgreSQL**: Uses SQLite for simpler deployment
 - ✅ **No Docker Socket**: DefaultRunLauncher runs jobs in same container
 - ✅ **Local Builds**: All images built locally (no Docker Hub dependency)
-- ✅ **Fewer Resources**: Reduced memory and complexity
+- ✅ **Fewer Resources**: Reduced memory and complexity with consolidated architecture
 
 ## Quick Start
 
@@ -26,38 +26,31 @@ Anomstack uses a **simplified Docker architecture** with local image building an
 make docker
 
 # Access services
-# - Dagster UI: http://localhost:3000
+# - Dagster UI: http://localhost:3000 (with embedded user code)
 # - Dashboard: http://localhost:5001
-# - PostgreSQL: localhost:5432 (if port forwarding enabled)
 ```
 
 ## Services
 
-### 1. Code Server (`anomstack_code`)
-- **Image**: Built locally from `docker/Dockerfile.anomstack_code`
-- **Purpose**: Runs Dagster user code via gRPC
-- **Port**: 4000 (internal)
-- **Restart Policy**: `always`
-- **Volumes**:
-  - `./tmp:/opt/dagster/app/tmp` (temporary files)
-  - `anomstack_metrics_duckdb:/data` (DuckDB data)
-  - `./dagster_home:/opt/dagster/dagster_home` (Dagster storage - includes SQLite)
-
-### 2. Dagster UI (`anomstack_webserver`)
+### 1. Dagster Webserver + User Code (`anomstack_webserver`)
 - **Image**: Built locally from `docker/Dockerfile.dagster`
-- **Purpose**: Web interface for pipeline management  
+- **Purpose**: Web interface and user code execution using direct Python module loading (no gRPC server needed)
 - **Port**: 3000 (external)
 - **Restart Policy**: `on-failure`
 - **Access**: http://localhost:3000
 - **Storage**: Uses SQLite files in mounted dagster_home volume
+- **Volumes**:
+  - `./tmp:/opt/dagster/app/tmp` (temporary files)
+  - `anomstack_metrics_duckdb:/data` (DuckDB data)
+  - `./dagster_home:/opt/dagster/dagster_home` (Dagster storage - includes SQLite)
 
-### 3. Dagster Daemon (`anomstack_daemon`) 
+### 2. Dagster Daemon (`anomstack_daemon`) 
 - **Image**: Built locally from `docker/Dockerfile.dagster`
 - **Purpose**: Background process for scheduling and run execution
 - **Restart Policy**: `on-failure`
 - **Run Launcher**: DefaultRunLauncher (runs jobs in same container process)
 
-### 4. Dashboard (`anomstack_dashboard`)
+### 3. Dashboard (`anomstack_dashboard`)
 - **Image**: Built locally from `docker/Dockerfile.anomstack_dashboard`
 - **Purpose**: FastHTML-based metrics dashboard
 - **Port**: 5001 (external) → 8080 (internal)
@@ -120,23 +113,22 @@ make docker-smart             # Build and start services (recommended)
 ### Debugging
 ```bash
 make docker-logs              # View all container logs
-make docker-logs-code         # View code server logs
-make docker-logs-dagit        # View Dagster UI logs
+make docker-logs-webserver    # View consolidated webserver logs (with embedded user code)
 make docker-logs-daemon       # View daemon logs
 make docker-logs-dashboard    # View dashboard logs
 ```
 
 ### Shell Access
 ```bash
-make docker-shell-code        # Shell into code server
-make docker-shell-dagit       # Shell into Dagster UI container
+make docker-shell-webserver   # Shell into consolidated webserver container
+make docker-shell-daemon      # Shell into daemon container
 make docker-shell-dashboard   # Shell into dashboard container
 ```
 
 ### Service Management
 ```bash
 make docker-restart-dashboard # Restart dashboard only
-make docker-restart-code      # Restart code server only
+make docker-restart-webserver # Restart webserver (with embedded user code) only
 ```
 
 ### Cleanup
@@ -197,9 +189,6 @@ docker run --rm -v anomstack_metrics_duckdb:/data -v $(pwd):/backup alpine tar c
 docker run --rm -v anomstack_metrics_duckdb:/data -v $(pwd):/backup alpine tar xzf /backup/duckdb-backup.tar.gz -C /
 ```
 
-### PostgreSQL Database
-- **Backup**: Use standard PostgreSQL backup tools
-- **Data**: Stored in Docker volume (not explicitly named)
 
 ## Networking
 
@@ -208,9 +197,8 @@ All services run on the `anomstack_network` bridge network:
 - **External access**: Only specific ports are exposed to host
 
 ### Port Mapping
-- **3000**: Dagster UI
+- **3000**: Dagster UI (with embedded user code)
 - **5001**: Dashboard
-- **5432**: PostgreSQL (if enabled)
 
 ## Troubleshooting
 
@@ -268,14 +256,15 @@ make docker-clean
 - Check for port conflicts (5001)
 
 #### Dagster UI Not Accessible
-- Verify dagit container is running
-- Check if code server is accessible
-- Review workspace.yaml configuration
+- Verify webserver container is running (now includes embedded user code)
+- Review workspace.yaml configuration for direct Python module loading
+- Check if user code is properly embedded in consolidated container
 
-#### Code Server Issues
-- Check if all dependencies are installed
+#### User Code Issues
+- Check if all dependencies are installed in consolidated container
 - Verify environment variables are set
 - Check DuckDB volume mount
+- Ensure Python module loading is working (no gRPC server needed)
 
 ## Production Considerations
 
@@ -297,8 +286,7 @@ make docker-clean
 ## Building Images
 
 All images are built locally from their respective Dockerfiles:
-- `anomstack_code_image` - Built from `docker/Dockerfile.anomstack_code`
-- `anomstack_dagster_image` - Built from `docker/Dockerfile.dagster`  
+- `anomstack_dagster_image` - Built from `docker/Dockerfile.dagster` (includes both webserver and user code)
 - `anomstack_dashboard_image` - Built from `docker/Dockerfile.anomstack_dashboard`
 
 ### Building Images