Skip to content

General dev work #178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jul 20, 2025
Merged

General dev work #178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
65849d5
feat: implement Dagster storage optimization to prevent disk buildup
andrewm4894 Jul 19, 2025
c340bd6
feat: enhance developer tooling with comprehensive Makefile targets a…
andrewm4894 Jul 19, 2025
7227589
docs: add storage optimization guide and improve README with Makefile…
andrewm4894 Jul 19, 2025
abf637d
chore: update documentation dependencies
andrewm4894 Jul 19, 2025
40800f7
feat: add utility scripts for Docker and Dagster management
andrewm4894 Jul 19, 2025
72d4af1
docs: expand README with new utility scripts and usage examples
andrewm4894 Jul 19, 2025
72b0f84
feat: Add configuration hot reload functionality
andrewm4894 Jul 19, 2025
40069a1
chore: Update Docker and deployment configurations
andrewm4894 Jul 19, 2025
58158bd
chore: Update build and environment configurations
andrewm4894 Jul 19, 2025
ab43ef7
docs: Update Python ingest example configuration
andrewm4894 Jul 19, 2025
f10ed33
fix: update cron schedules in python_ingest_simple.yaml
andrewm4894 Jul 20, 2025
518d1fd
chore: update environment and Docker configurations
andrewm4894 Jul 20, 2025
b38f0bf
fix: refine environment variable override logic in config
andrewm4894 Jul 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 28 additions & 7 deletions .example.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ ANOMSTACK_MODEL_PATH=local://./tmp/models
# for s3 bucket
# ANOMSTACK_MODEL_PATH=s3://your-bucket/models

# home directory for anomstack (defaults to current working directory)
# used for docker volume mounts in dagster_docker.yaml
ANOMSTACK_HOME=.

# slack bot token for alerting
# ANOMSTACK_SLACK_BOT_TOKEN=
# slack channel for alerting
Expand All @@ -66,13 +70,14 @@ ANOMSTACK_ALERT_EMAIL_PASSWORD=
# ANOMSTACK_FAILURE_EMAIL_SMTP_PORT=587
# ANOMSTACK_FAILURE_EMAIL_PASSWORD=

# dagster related env vars
ANOMSTACK_DAGSTER_LOCAL_ARTIFACT_STORAGE_DIR=tmp
ANOMSTACK_DAGSTER_OVERALL_CONCURRENCY_LIMIT=10
# dagster related env vars - LIGHTWEIGHT DEFAULTS
# Use smaller, more manageable directories to prevent disk space issues
ANOMSTACK_DAGSTER_LOCAL_ARTIFACT_STORAGE_DIR=tmp_light/artifacts
ANOMSTACK_DAGSTER_OVERALL_CONCURRENCY_LIMIT=5 # Reduced from 10
ANOMSTACK_DAGSTER_DEQUEUE_USE_THREADS=false
ANOMSTACK_DAGSTER_DEQUEUE_NUM_WORKERS=4
ANOMSTACK_DAGSTER_LOCAL_COMPUTE_LOG_MANAGER_DIRECTORY=tmp
ANOMSTACK_DAGSTER_SQLITE_STORAGE_BASE_DIR=tmp
ANOMSTACK_DAGSTER_DEQUEUE_NUM_WORKERS=2 # Reduced from 4
ANOMSTACK_DAGSTER_LOCAL_COMPUTE_LOG_MANAGER_DIRECTORY=tmp_light/compute_logs
ANOMSTACK_DAGSTER_SQLITE_STORAGE_BASE_DIR=tmp_light/storage

# OpenAI env vars for LLM based alerts
ANOMSTACK_OPENAI_KEY=
Expand Down Expand Up @@ -131,4 +136,20 @@ POSTHOG_API_KEY=

# example to enable some jobs via env vars
# ANOMSTACK__PYTHON_INGEST_SIMPLE__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__PLOT_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__LLMALERT_DEFAULT_SCHEDULE_STATUS=RUNNING
# ANOMSTACK__PYTHON_INGEST_SIMPLE__DELETE_DEFAULT_SCHEDULE_STATUS=RUNNING

# automatic configuration reloading
# Enable scheduled config reload job (runs every N minutes)
ANOMSTACK_AUTO_CONFIG_RELOAD=false
ANOMSTACK_CONFIG_RELOAD_CRON="*/5 * * * *"
ANOMSTACK_CONFIG_RELOAD_STATUS=STOPPED

# Enable smart config file watcher sensor (reloads only when files change)
ANOMSTACK_CONFIG_WATCHER=true
ANOMSTACK_CONFIG_WATCHER_INTERVAL=30
Comment on lines +147 to +155
Copy link
Contributor

@coderabbitai coderabbitai bot Jul 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Watcher variable name mismatch will disable the feature completely.

Code & docs previously referenced ANOMSTACK_CONFIG_WATCHER_POLL_INTERVAL,
but the file introduces ANOMSTACK_CONFIG_WATCHER_INTERVAL.

-ANOMSTACK_CONFIG_WATCHER_INTERVAL=30
+ANOMSTACK_CONFIG_WATCHER_POLL_INTERVAL=30   # keep in sync with anomstack/config.py

Search the repo for the constant to confirm.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# automatic configuration reloading
# Enable scheduled config reload job (runs every N minutes)
ANOMSTACK_AUTO_CONFIG_RELOAD=false
ANOMSTACK_CONFIG_RELOAD_CRON="*/5 * * * *"
ANOMSTACK_CONFIG_RELOAD_STATUS=STOPPED
# Enable smart config file watcher sensor (reloads only when files change)
ANOMSTACK_CONFIG_WATCHER=true
ANOMSTACK_CONFIG_WATCHER_INTERVAL=30
# automatic configuration reloading
# Enable scheduled config reload job (runs every N minutes)
ANOMSTACK_AUTO_CONFIG_RELOAD=false
ANOMSTACK_CONFIG_RELOAD_CRON="*/5 * * * *"
ANOMSTACK_CONFIG_RELOAD_STATUS=STOPPED
# Enable smart config file watcher sensor (reloads only when files change)
ANOMSTACK_CONFIG_WATCHER=true
ANOMSTACK_CONFIG_WATCHER_POLL_INTERVAL=30 # keep in sync with anomstack/config.py
🤖 Prompt for AI Agents 
In the .example.env file around lines 143 to 151, the environment variable for
the watcher interval is named ANOMSTACK_CONFIG_WATCHER_INTERVAL, but the code
and documentation expect ANOMSTACK_CONFIG_WATCHER_POLL_INTERVAL. Rename
ANOMSTACK_CONFIG_WATCHER_INTERVAL to ANOMSTACK_CONFIG_WATCHER_POLL_INTERVAL to
match the expected variable name and ensure the watcher feature works correctly.
Verify by searching the repo for all references to this constant to confirm
consistency.

102 changes: 98 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ dev:

.PHONY: docker docker-dev docker-smart docker-build docker-dev-build docker-tag docker-push docker-build-push
.PHONY: docker-pull docker-clean docker-logs docker-logs-code docker-logs-dagit docker-logs-daemon docker-logs-dashboard
.PHONY: docker-shell-code docker-shell-dagit docker-shell-dashboard docker-restart-dashboard docker-restart-code
.PHONY: docker-shell-code docker-shell-dagit docker-shell-dashboard docker-restart-dashboard docker-restart-code docker-restart reload-config enable-auto-reload enable-config-watcher
.PHONY: docker-stop docker-down docker-rm docker-prune

# start docker containers (now uses pre-built images)
Expand Down Expand Up @@ -124,6 +124,28 @@ docker-restart-dashboard:
docker-restart-code:
docker compose restart anomstack_code

# restart all containers (useful for .env changes)
docker-restart:
docker compose restart

# reload configuration without restarting containers (hot reload)
reload-config:
@echo "🔄 Reloading Anomstack configuration..."
python3 scripts/reload_config.py

# enable automatic config reloading via Dagster scheduled job
enable-auto-reload:
@echo "🤖 Enabling automatic configuration reloading..."
@echo "ANOMSTACK_AUTO_CONFIG_RELOAD=true" >> .env
@echo "ANOMSTACK_CONFIG_RELOAD_STATUS=RUNNING" >> .env
@echo "✅ Auto reload enabled! Restart containers: make docker-restart"

# enable smart config file watcher sensor
enable-config-watcher:
@echo "👁️ Enabling smart configuration file watcher..."
@echo "ANOMSTACK_CONFIG_WATCHER=true" >> .env
@echo "✅ Config watcher enabled! Restart containers: make docker-restart"

# alias for docker-stop
docker-down:
docker compose down
Expand All @@ -137,6 +159,58 @@ docker-prune:
docker compose down -v --remove-orphans
docker system prune -a -f

# =============================================================================
# RESET OPERATIONS
# =============================================================================

.PHONY: reset-gentle reset-medium reset-nuclear reset-full-nuclear reset-interactive

# interactive reset with guided options
reset-interactive:
@scripts/utils/reset_docker.sh

# gentle reset: rebuild containers with fresh images (safest)
reset-gentle:
@scripts/utils/reset_docker.sh gentle

# medium reset: remove containers, keep data volumes
reset-medium:
@scripts/utils/reset_docker.sh medium

# nuclear reset: remove everything including local data
reset-nuclear:
@scripts/utils/reset_docker.sh nuclear

# full nuclear reset: nuclear + full docker system cleanup (maximum cleanup)
reset-full-nuclear:
@scripts/utils/reset_docker.sh full-nuclear

# =============================================================================
# DAGSTER STORAGE CLEANUP
# =============================================================================

.PHONY: dagster-cleanup-status dagster-cleanup-minimal dagster-cleanup-standard dagster-cleanup-aggressive

# show current dagster storage usage and configuration status
dagster-cleanup-status:
@scripts/utils/cleanup_dagster_storage.sh status

# minimal dagster cleanup - remove old logs only (safe)
dagster-cleanup-minimal:
@scripts/utils/cleanup_dagster_storage.sh minimal

# standard dagster cleanup - remove runs older than 30 days
dagster-cleanup-standard:
@scripts/utils/cleanup_dagster_storage.sh standard

# aggressive dagster cleanup - remove runs older than 7 days
dagster-cleanup-aggressive:
@scripts/utils/cleanup_dagster_storage.sh aggressive

# interactive dagster cleanup menu
dagster-cleanup-menu:
@scripts/utils/cleanup_dagster_storage.sh menu

# =============================================================================
# DASHBOARD OPERATIONS
# =============================================================================
Expand Down Expand Up @@ -185,11 +259,31 @@ coverage:
# DOCUMENTATION
# =============================================================================

.PHONY: docs
.PHONY: docs docs-start docs-build docs-serve docs-clear docs-install

# start documentation development server
# start documentation development server (alias for docs-start)
docs:
cd docs && yarn start
@$(MAKE) docs-start

# install documentation dependencies
docs-install:
cd docs && npm install

# start development server with live reload
docs-start:
cd docs && npm start

# build static documentation site
docs-build:
cd docs && npm run build

# serve built documentation locally
docs-serve:
cd docs && npm run serve

# clear documentation build cache
docs-clear:
cd docs && npm run clear

# =============================================================================
# DEPENDENCIES
Expand Down
Loading