Merge branch 'main' into config-via-env

Author: andrewm4894

.devcontainer/README.md

@@ -0,0 +1,81 @@
+# Devcontainer Configurations
+
+This directory contains multiple devcontainer configurations for different use cases.
+
+## 📋 Available Configurations
+
+### 1. `devcontainer.json` - **User/Demo** (Default)
+- **Purpose**: Quick start for new users or demos
+- **Docker Strategy**: Pulls pre-built images from Docker Hub
+- **Speed**: ⚡ Fast startup (30-60 seconds)
+- **Use Cases**: 
+  - New users trying out anomstack
+  - Demos and presentations
+  - Quick testing without development setup
+
+### 2. `devcontainer.dev.json` - **Development**
+- **Purpose**: Full development environment
+- **Docker Strategy**: Builds images locally with source code changes
+- **Speed**: 🐌 Slower startup (3-5 minutes)
+- **Use Cases**:
+  - Active development on anomstack
+  - Testing code changes
+  - Contributing to the project
+
+## 🚀 How to Use
+
+### GitHub Codespaces
+
+1. **Default (User/Demo)**: Just click "Create codespace" - uses `devcontainer.json`
+2. **Development**: When creating a codespace, click "..." → "Configure dev container" → select `devcontainer.dev.json`
+
+### VS Code Dev Containers
+
+1. **Default**: Open folder in VS Code → "Reopen in Container"
+2. **Development**: 
+   - Open Command Palette (`Ctrl+Shift+P`)
+   - "Dev Containers: Reopen in Container"
+   - Select `devcontainer.dev.json`
+
+### Manual Selection
+
+You can also rename the files to switch defaults:
+```bash
+# Switch to development as default
+mv devcontainer.json devcontainer.user.json
+mv devcontainer.dev.json devcontainer.json
+```
+
+## 🏗️ What Each Configuration Does
+
+### Both Configurations
+- Set up VS Code extensions for Python, Docker, and development
+- Forward ports: 3000 (Dagster), 5001 (Dashboard), 5432 (PostgreSQL)
+- Run initialization and post-create scripts
+- Enable Docker-in-Docker support
+
+### Key Differences
+| Feature | User/Demo | Development |
+|---------|-----------|-------------|
+| Docker Images | Pull from Hub | Build locally |
+| Startup Time | ~60 seconds | ~5 minutes |
+| Code Changes | View only | Full development |
+| Extensions | Basic set | Extended dev tools |
+
+## 🎯 Recommendations
+
+- **First time users**: Use default `devcontainer.json`
+- **Regular development**: Use `devcontainer.dev.json`
+- **Demos/presentations**: Use default `devcontainer.json`
+- **Contributing**: Use `devcontainer.dev.json`
+
+## 🔧 Services Available
+
+Both configurations provide:
+- **Dagster Webserver**: [http://localhost:3000](http://localhost:3000)
+- **Anomstack Dashboard**: [http://localhost:5001](http://localhost:5001)
+- **PostgreSQL Database**: `localhost:5432`
+
+## 📚 Documentation
+
+For more information, see the main [README.md](../README.md) and [DOCKER.md](../DOCKER.md) files. 

.devcontainer/devcontainer.dev.json

@@ -0,0 +1,50 @@
+{
+    "name": "anomstack (Development)",
+    "dockerComposeFile": [
+        "../docker-compose.yaml",
+        "../docker-compose.dev.yaml"
+    ],
+    "service": "anomstack_code",
+    "workspaceFolder": "/opt/dagster/app",
+    "forwardPorts": [3000, 5001, 5432],
+    "portsAttributes": {
+        "3000": {
+            "label": "Dagster Webserver",
+            "onAutoForward": "notify"
+        },
+        "5001": {
+            "label": "Dashboard",
+            "onAutoForward": "notify"
+        },
+        "5432": {
+            "label": "PostgreSQL",
+            "onAutoForward": "silent"
+        }
+    },
+    "remoteUser": "root",
+    "initializeCommand": "bash .devcontainer/initialize.sh",
+    "postCreateCommand": [
+        "bash",
+        "-c",
+        "chmod +x .devcontainer/post_create_command.sh && bash .devcontainer/post_create_command.sh"
+    ],
+    "customizations": {
+        "vscode": {
+            "extensions": [
+                "github.copilot",
+                "github.copilot-chat",
+                "ms-azuretools.vscode-docker",
+                "ms-vscode.makefile-tools",
+                "ms-python.python",
+                "ms-python.pylint",
+                "ms-python.black-formatter",
+                "ms-python.isort",
+                "ms-python.flake8"
+            ]
+        }
+    },
+    "features": {
+        "ghcr.io/devcontainers/features/docker-in-docker:2": {}
+    },
+    "shutdownAction": "stopCompose"
+} 

.devcontainer/devcontainer.json

@@ -1,5 +1,5 @@
 {
-    "name": "anomstack",
+    "name": "anomstack (User/Demo)",
     "dockerComposeFile": "../docker-compose.yaml",
     "service": "anomstack_code",
     "workspaceFolder": "/opt/dagster/app",

.devcontainer/initialize.sh

@@ -18,6 +18,37 @@ mkdir -p tmp
 # Create dagster_home directory at the same path used by Docker Compose
 mkdir -p /opt/dagster/dagster_home
 
+# Set python_ingest_simple example jobs to RUNNING for GitHub Codespaces
+PYTHON_INGEST_CONFIG="metrics/examples/python/python_ingest_simple/python_ingest_simple.yaml"
+if [ -f "$PYTHON_INGEST_CONFIG" ]; then
+    echo "Updating python_ingest_simple job schedules to RUNNING for Codespaces..."
+    
+    # Array of schedule statuses to set to RUNNING
+    SCHEDULE_STATUSES=(
+        "ingest_default_schedule_status"
+        "train_default_schedule_status"
+        "score_default_schedule_status"
+        "alert_default_schedule_status"
+    )
+    
+    # Update each schedule status
+    for status in "${SCHEDULE_STATUSES[@]}"; do
+        if grep -q "$status:" "$PYTHON_INGEST_CONFIG"; then
+            # Replace existing line
+            sed -i "s/${status}:.*/${status}: \"RUNNING\"/" "$PYTHON_INGEST_CONFIG"
+            echo "Updated existing $status to RUNNING"
+        else
+            # Add new line after metric_batch line
+            sed -i "/^metric_batch:/a ${status}: \"RUNNING\"" "$PYTHON_INGEST_CONFIG"
+            echo "Added $status: RUNNING"
+        fi
+    done
+    
+    echo "Updated python_ingest_simple job schedules to RUNNING"
+else
+    echo "Warning: python_ingest_simple.yaml not found at $PYTHON_INGEST_CONFIG"
+fi
+
 # Any other pre-container setup can go here
 # For example:
 # - Download required files

.example.env

@@ -95,11 +95,12 @@ ANOMSTACK_LLM_PLATFORM=openai
 # some dagster env vars
 DAGSTER_LOG_LEVEL=DEBUG
 DAGSTER_CONCURRENCY=4
-DAGSTER_HOME=/opt/dagster/dagster_home
 
 # max runtime for a job in dagster
 # https://docs.dagster.io/deployment/run-monitoring#general-run-timeouts
 ANOMSTACK_MAX_RUNTIME_SECONDS_TAG=3600
+# kill runs that exceed this many minutes
+ANOMSTACK_KILL_RUN_AFTER_MINUTES=60
 
 # postgres related env vars
 ANOMSTACK_POSTGRES_USER=postgres_user
@@ -126,4 +127,4 @@ ANOMSTACK_CLICKHOUSE_DATABASE=default
 ANOMSTACK_DASHBOARD_PORT = 5001
 
 # PostHog API key for analytics
-POSTHOG_API_KEY=
+POSTHOG_API_KEY=

CONTRIBUTING.md

@@ -146,7 +146,7 @@ We want your work to be readable by others; therefore, we encourage you to note
 - All new code should include appropriate tests. See the [`tests/README.md`](./tests/README.md) for detailed testing guidelines.
 - Run the full test suite before submitting: `pytest tests/`
 - Check test coverage: `pytest tests/ --cov=anomstack --cov-report=term-missing`
-- The project currently maintains **64% test coverage**. Contributions that increase coverage are especially welcome!
+- The project currently maintains **63% test coverage**. Contributions that increase coverage are especially welcome!
 
 #### Updating the Coverage Badge
 

Makefile

@@ -6,9 +6,9 @@ SHELL=/bin/bash
 
 .PHONY: local locald kill-locald ps-locald dev
 
-# start dagster locally
+# start dagster locally (simple - just set DAGSTER_HOME directly)
 local:
-	dagster dev -f anomstack/main.py
+	DAGSTER_HOME=$$(pwd)/dagster_home dagster dev -f anomstack/main.py
 
 # start dagster locally as a daemon with no log file
 locald:
@@ -30,7 +30,7 @@ dev:
 # DOCKER OPERATIONS
 # =============================================================================
 
-.PHONY: docker docker-dev docker-build docker-dev-build docker-tag docker-push docker-build-push
+.PHONY: docker docker-dev docker-smart docker-build docker-dev-build docker-tag docker-push docker-build-push
 .PHONY: docker-pull docker-clean docker-logs docker-logs-code docker-logs-dagit docker-logs-daemon docker-logs-dashboard
 .PHONY: docker-shell-code docker-shell-dagit docker-shell-dashboard docker-restart-dashboard docker-restart-code
 .PHONY: docker-stop docker-down docker-rm docker-prune
@@ -39,6 +39,17 @@ dev:
 docker:
 	docker compose up -d
 
+# smart docker start: try to pull, fallback to build if images don't exist
+docker-smart:
+	@echo "🔄 Attempting to pull pre-built images..."
+	@if docker compose pull 2>/dev/null; then \
+		echo "✅ Successfully pulled images, starting containers..."; \
+		docker compose up -d; \
+	else \
+		echo "⚠️  Pull failed, building images locally..."; \
+		make docker-dev-build && make docker-dev; \
+	fi
+
 # start docker containers with local development images
 docker-dev:
 	docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d
@@ -113,10 +124,6 @@ docker-restart-dashboard:
 docker-restart-code:
 	docker compose restart anomstack_code
 
-# stop all containers
-docker-stop:
-	docker compose down
-
 # alias for docker-stop
 docker-down:
 	docker compose down
@@ -202,8 +209,20 @@ requirements-install:
 # UTILITIES
 # =============================================================================
 
-.PHONY: posthog-example
+.PHONY: posthog-example kill-long-runs
 
 # run the PostHog example ingest function
 posthog-example:
 	python scripts/posthog_example.py
+
+# kill any dagster runs exceeding configured timeout
+kill-long-runs:
+	python scripts/kill_long_running_tasks.py
+
+# run docker in dev mode with correct environment
+docker-dev-env:
+	docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d
+
+# stop docker containers
+docker-stop:
+	docker compose -f docker-compose.yaml -f docker-compose.dev.yaml down

README.md

@@ -13,7 +13,7 @@
 <a href="https://github.com/andrewm4894/anomstack/blob/main/LICENSE">![License](https://img.shields.io/badge/License-MIT-yellow.svg)</a>
 <a href="https://github.com/andrewm4894/anomstack/actions/workflows/pytest.yaml">![GitHub PyTest Workflow Status](https://img.shields.io/github/actions/workflow/status/andrewm4894/anomstack/pytest.yaml?label=Tests)</a>
 <a href="https://github.com/andrewm4894/anomstack/actions/workflows/pre-commit.yaml">![GitHub Pre-Commit Workflow Status](https://img.shields.io/github/actions/workflow/status/andrewm4894/anomstack/pre-commit.yaml?label=Pre-Commit)</a>
-<a href="./tests/README.md#coverage-report">![Coverage](https://img.shields.io/badge/Coverage-64%25-yellow?logo=pytest)</a>
+<a href="./tests/README.md#coverage-report">![Coverage](https://img.shields.io/badge/Coverage-63%25-yellow?logo=pytest)</a>
 
 </div>
 
@@ -654,6 +654,18 @@ Below you see an example of an LLM alert via email. In this case we add a descri
 
 </details>
 
+## Killing Long Running Jobs
+
+Sometimes Dagster runs can get stuck. Anomstack ships with a sensor that
+terminates any run exceeding a configurable timeout. By default runs are killed
+after 60 minutes. You can override this in your `dagster.yaml` or via the
+`ANOMSTACK_KILL_RUN_AFTER_MINUTES` environment variable. You can also invoke the
+cleanup manually with:
+
+```bash
+make kill-long-runs
+```
+
 ## Contributing
 
 Read the [contributing guide](./CONTRIBUTING.md) to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Anomstack.

anomstack/main.py

@@ -14,6 +14,7 @@
 from anomstack.jobs.summary import summary_jobs, summary_schedules
 from anomstack.jobs.train import train_jobs, train_schedules
 from anomstack.sensors.failure import email_on_run_failure
+from anomstack.sensors.timeout import kill_long_running_runs
 
 jobs = (
     ingest_jobs
@@ -26,7 +27,7 @@
     + summary_jobs
     + delete_jobs
 )
-sensors = [email_on_run_failure]
+sensors = [email_on_run_failure, kill_long_running_runs]
 schedules = (
     ingest_schedules
     + train_schedules