From 3450e252bc98ee779087e5f6e127a3d127a43edb Mon Sep 17 00:00:00 2001 From: andrewm4894 Date: Tue, 22 Jul 2025 16:25:30 +0100 Subject: [PATCH 1/3] feat: add deployment profiles system - Add profiles/ directory with demo, production, and development profiles - Enhance deploy_fly.sh with --profile flag support - Add profile merging with .env precedence (secrets preserved) - Support ANOMSTACK____ environment variable overrides - Enable environment-specific configurations without modifying examples --- profiles/README.md | 182 ++++++++++++++++++++++ profiles/demo.env | 52 +++++++ profiles/development.env | 19 +++ profiles/production.env | 10 ++ scripts/deployment/deploy_fly.sh | 116 +++++++++++--- scripts/deployment/preview_fly_secrets.sh | 65 +++++++- 6 files changed, 421 insertions(+), 23 deletions(-) create mode 100644 profiles/README.md create mode 100644 profiles/demo.env create mode 100644 profiles/development.env create mode 100644 profiles/production.env diff --git a/profiles/README.md b/profiles/README.md new file mode 100644 index 00000000..5951b675 --- /dev/null +++ b/profiles/README.md @@ -0,0 +1,182 @@ +# Anomstack Deployment Profiles + +This directory contains deployment profiles that act like "Helm values files" for Anomstack, allowing you to configure different environments without modifying the original metric batch examples. + +## How It Works + +Deployment profiles leverage Anomstack's existing environment variable override system (`ANOMSTACK____`) to enable and configure metric batches at deployment time. + +### Benefits + +✅ **Keep examples pristine** - Original examples remain as clean templates +✅ **Environment-specific configs** - Different settings for demo, dev, production +✅ **Easy deployment** - Apply profiles during deployment with a single flag +✅ **Source controlled** - All configurations are versioned and visible +✅ **User-friendly** - Others can easily redeploy your exact demo configuration +✅ **Hot-reloadable** - Changes to profiles can be applied without restarts + +## Available Profiles + +### `demo.env` +**Purpose**: Configuration for the public demo instance +**Features**: +- Enables key examples (netdata, weather, hackernews, python_ingest_simple) +- All job schedules running +- Email alerts enabled +- Demo-appropriate settings + +### `production.env` +**Purpose**: Production deployment template +**Features**: +- Disables examples for clean deployment +- Conservative alert thresholds +- Production-grade database paths +- Team-based alert routing + +### `development.env` +**Purpose**: Development/staging environments +**Features**: +- Enables all examples for testing +- More sensitive alert thresholds +- Hot-reload features enabled +- Development-friendly settings + +## Usage + +### Option 1: Enhanced Deployment Script (Recommended) + +```bash +# Deploy with demo profile +./scripts/deployment/deploy_fly.sh --profile demo + +# Deploy with production profile +./scripts/deployment/deploy_fly.sh --profile production + +# Deploy with custom profile +./scripts/deployment/deploy_fly.sh --profile my-custom-profile +``` + +### Option 2: Manual Profile Application + +```bash +# Copy profile to .env and deploy +cp profiles/demo.env .env +# Add your secrets to .env +echo "ANOMSTACK_OPENAI_KEY=your-key" >> .env +./scripts/deployment/deploy_fly.sh +``` + +### Option 3: Profile Merging + +```bash +# Start with your base .env +cp .example.env .env +# Edit with your API keys, credentials, etc. +vim .env + +# Merge with demo profile for deployment +cat profiles/demo.env >> .env +./scripts/deployment/deploy_fly.sh +``` + +## Creating Custom Profiles + +### 1. Create Profile File + +```bash +# Create new profile +vim profiles/my-company.env +``` + +### 2. Profile Structure + +```bash +# Company Production Anomstack Profile +ANOMSTACK_IGNORE_EXAMPLES=yes +ANOMSTACK_TABLE_KEY=company_production.metrics + +# Enable your specific metric batches +ANOMSTACK__MY_BUSINESS_METRICS__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__MY_BUSINESS_METRICS__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__MY_BUSINESS_METRICS__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__MY_BUSINESS_METRICS__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__MY_BUSINESS_METRICS__ALERT_METHODS=email,slack +``` + +### 3. Deploy with Custom Profile + +```bash +./scripts/deployment/deploy_fly.sh --profile my-company +``` + +## Profile Composition + +Profiles are applied on top of your base `.env` configuration: + +1. **Base `.env`**: Your secrets, API keys, database credentials +2. **Profile**: Environment-specific metric batch enablement and configuration +3. **Deployment**: Combined configuration deployed as Fly secrets + +This layered approach means: +- **Secrets stay in your `.env`** (not committed to source control) +- **Profile configurations are shareable** and source controlled +- **Others can use your exact demo setup** by applying your profile + +## Environment Variable Override System + +Profiles use Anomstack's built-in override system: + +```bash +# Format: ANOMSTACK____= +ANOMSTACK__NETDATA__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__WEATHER__ALERT_METHODS=email,slack +ANOMSTACK__PYTHON_INGEST_SIMPLE__ALERT_THRESHOLD=0.7 +``` + +This allows you to: +- Enable/disable specific metric batches +- Customize schedules, alert methods, thresholds +- Override any parameter from `metrics/defaults/defaults.yaml` +- Configure multiple batches consistently + +## Examples + +### Demo Instance Recreation + +Anyone can recreate your exact demo instance: + +```bash +git clone https://github.com/andrewm4894/anomstack.git +cd anomstack +cp .example.env .env +# Add their own API keys to .env +./scripts/deployment/deploy_fly.sh --profile demo my-demo-instance +``` + +### Multi-Environment Deployment + +```bash +# Deploy to different environments with different profiles +./scripts/deployment/deploy_fly.sh --profile development anomstack-dev +./scripts/deployment/deploy_fly.sh --profile production anomstack-prod +./scripts/deployment/deploy_fly.sh --profile demo anomstack-demo +``` + +### Custom Company Setup + +```bash +# Create company-specific profile +cat > profiles/acme-corp.env << EOF +ANOMSTACK_IGNORE_EXAMPLES=yes +ANOMSTACK_TABLE_KEY=acme.anomaly_detection.metrics +ANOMSTACK__ACME_SALES_METRICS__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__ACME_USER_METRICS__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +EOF + +# Deploy with company profile +./scripts/deployment/deploy_fly.sh --profile acme-corp acme-anomstack +``` + +## Next Steps + +To implement profile support in the deployment script, see the enhancement in `scripts/deployment/deploy_fly.sh` that adds the `--profile` flag. \ No newline at end of file diff --git a/profiles/demo.env b/profiles/demo.env new file mode 100644 index 00000000..cbea558f --- /dev/null +++ b/profiles/demo.env @@ -0,0 +1,52 @@ +# Anomstack Demo Deployment Profile +# This profile enables specific metric batches for the demo instance +# Apply this during deployment to get the demo configuration + +# ============================================================================= +# DEMO PROFILE +# ============================================================================= + +# Enable examples for demo (override default behavior) +ANOMSTACK_IGNORE_EXAMPLES=no + +# Store metrics in local duckdb volume +ANOMSTACK_DUCKDB_PATH=/data/anomstack.db + +# Store models on local data +ANOMSTACK_MODEL_PATH=local:///data/models + +# max runtime for a job in dagster +# https://docs.dagster.io/deployment/run-monitoring#general-run-timeouts +ANOMSTACK_MAX_RUNTIME_SECONDS_TAG=3600 + +# Enable Netdata +ANOMSTACK__NETDATA__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__LLMALERT_DEFAULT_SCHEDULE_STATUS=STOPPED + +# Enable Netdata Httpcheck +ANOMSTACK__NETDATA_HTTPCHECK__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA_HTTPCHECK__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA_HTTPCHECK__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA_HTTPCHECK__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA_HTTPCHECK__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA_HTTPCHECK__LLMALERT_DEFAULT_SCHEDULE_STATUS=STOPPED + +# Enable Github +ANOMSTACK__GITHUB__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__GITHUB__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__GITHUB__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__GITHUB__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__GITHUB__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__GITHUB__LLMALERT_DEFAULT_SCHEDULE_STATUS=STOPPED + +# Enable Currency +ANOMSTACK__CURRENCY__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__CURRENCY__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__CURRENCY__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__CURRENCY__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__CURRENCY__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__CURRENCY__LLMALERT_DEFAULT_SCHEDULE_STATUS=STOPPED diff --git a/profiles/development.env b/profiles/development.env new file mode 100644 index 00000000..26d2b481 --- /dev/null +++ b/profiles/development.env @@ -0,0 +1,19 @@ +# Anomstack Development Deployment Profile +# This profile configures Anomstack for development/staging environments +# Apply this during deployment for development work + +# ============================================================================= +# DEVELOPMENT PROFILE +# ============================================================================= + +# Enable examples for demo (override default behavior) +ANOMSTACK_IGNORE_EXAMPLES=no + +# Enable and configure netdata metric batch for demo +ANOMSTACK__NETDATA__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__LLMALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__PLOT_DEFAULT_SCHEDULE_STATUS=RUNNING \ No newline at end of file diff --git a/profiles/production.env b/profiles/production.env new file mode 100644 index 00000000..f6086cd3 --- /dev/null +++ b/profiles/production.env @@ -0,0 +1,10 @@ +# Anomstack Production Deployment Profile +# This profile configures Anomstack for production usage +# Apply this during deployment for production environments + +# ============================================================================= +# PRODUCTION PROFILE +# ============================================================================= + +# Disable examples in production for cleaner deployment +ANOMSTACK_IGNORE_EXAMPLES=yes diff --git a/scripts/deployment/deploy_fly.sh b/scripts/deployment/deploy_fly.sh index 28e9f3a3..53eed853 100755 --- a/scripts/deployment/deploy_fly.sh +++ b/scripts/deployment/deploy_fly.sh @@ -5,6 +5,32 @@ set -e echo "🚀 Deploying Anomstack to Fly.io..." +# Parse command line arguments for profile support +PROFILE="" +APP_NAME="" + +while [[ $# -gt 0 ]]; do + case $1 in + --profile) + PROFILE="$2" + shift 2 + ;; + -p) + PROFILE="$2" + shift 2 + ;; + *) + if [[ -z "$APP_NAME" ]]; then + APP_NAME="$1" + fi + shift + ;; + esac +done + +# Set default app name if not provided +APP_NAME="${APP_NAME:-anomstack-demo}" + # Check if fly CLI is installed and user is logged in if ! command -v fly &> /dev/null; then echo "❌ Fly CLI is not installed. Please install it first:" @@ -18,11 +44,45 @@ if ! fly auth whoami &> /dev/null; then fi echo "✅ Fly CLI is installed and you are logged in." - -# Set app name (can be customized) -APP_NAME="${1:-anomstack-demo}" echo "📱 App name: $APP_NAME" +# Handle deployment profile if specified +if [[ -n "$PROFILE" ]]; then + PROFILE_FILE="profiles/${PROFILE}.env" + if [[ -f "$PROFILE_FILE" ]]; then + echo "🎯 Using deployment profile: $PROFILE" + echo "📄 Profile file: $PROFILE_FILE" + + # Create temporary merged .env file + TEMP_ENV_FILE=$(mktemp) + + # Start with existing .env if it exists + if [[ -f ".env" ]]; then + cat ".env" > "$TEMP_ENV_FILE" + echo "✅ Base configuration loaded from .env" + else + touch "$TEMP_ENV_FILE" + fi + + # Append profile configuration (profile values override .env values) + echo "" >> "$TEMP_ENV_FILE" # Add separator + echo "# Profile: $PROFILE (applied during deployment)" >> "$TEMP_ENV_FILE" + cat "$PROFILE_FILE" >> "$TEMP_ENV_FILE" + echo "✅ Profile configuration merged" + + # Use the merged file for deployment + ENV_FILE="$TEMP_ENV_FILE" + else + echo "❌ Profile file not found: $PROFILE_FILE" + echo "Available profiles:" + ls -1 profiles/*.env 2>/dev/null | sed 's/profiles\///g' | sed 's/\.env//g' | sed 's/^/ - /' + exit 1 + fi +else + ENV_FILE=".env" + echo "📄 Using standard .env file" +fi + # Create Fly.io app if it doesn't exist @@ -65,24 +125,24 @@ declare -a all_secrets=() all_secrets+=("DAGSTER_HOME=/opt/dagster/dagster_home") all_secrets+=("PYTHONPATH=/opt/dagster/app") -# Set Fly.io defaults only if not already defined in .env -if ! grep -q "^DAGSTER_CODE_SERVER_HOST=" ".env" 2>/dev/null; then +# Set Fly.io defaults only if not already defined in configuration +if ! grep -q "^DAGSTER_CODE_SERVER_HOST=" "$ENV_FILE" 2>/dev/null; then all_secrets+=("DAGSTER_CODE_SERVER_HOST=localhost") fi -if ! grep -q "^ANOMSTACK_DUCKDB_PATH=" ".env" 2>/dev/null; then +if ! grep -q "^ANOMSTACK_DUCKDB_PATH=" "$ENV_FILE" 2>/dev/null; then all_secrets+=("ANOMSTACK_DUCKDB_PATH=/data/anomstack.db") fi -if ! grep -q "^ANOMSTACK_MODEL_PATH=" ".env" 2>/dev/null; then +if ! grep -q "^ANOMSTACK_MODEL_PATH=" "$ENV_FILE" 2>/dev/null; then all_secrets+=("ANOMSTACK_MODEL_PATH=local:///data/models") fi -if ! grep -q "^ANOMSTACK_TABLE_KEY=" ".env" 2>/dev/null; then +if ! grep -q "^ANOMSTACK_TABLE_KEY=" "$ENV_FILE" 2>/dev/null; then all_secrets+=("ANOMSTACK_TABLE_KEY=metrics") fi -if ! grep -q "^ANOMSTACK_IGNORE_EXAMPLES=" ".env" 2>/dev/null; then +if ! grep -q "^ANOMSTACK_IGNORE_EXAMPLES=" "$ENV_FILE" 2>/dev/null; then all_secrets+=("ANOMSTACK_IGNORE_EXAMPLES=no") fi @@ -90,8 +150,8 @@ fi ADMIN_USERNAME="admin" ADMIN_PASSWORD="" -if [[ ! -f ".env" ]] || ! grep -q "ANOMSTACK_ADMIN_PASSWORD=" ".env" 2>/dev/null; then - echo "🔐 Generating admin credentials (no .env admin password found)..." +if [[ ! -f "$ENV_FILE" ]] || ! grep -q "ANOMSTACK_ADMIN_PASSWORD=" "$ENV_FILE" 2>/dev/null; then + echo "🔐 Generating admin credentials (no admin password found in config)..." ADMIN_PASSWORD="$(openssl rand -base64 12)" all_secrets+=("ANOMSTACK_ADMIN_USERNAME=$ADMIN_USERNAME") all_secrets+=("ANOMSTACK_ADMIN_PASSWORD=$ADMIN_PASSWORD") @@ -100,18 +160,18 @@ if [[ ! -f ".env" ]] || ! grep -q "ANOMSTACK_ADMIN_PASSWORD=" ".env" 2>/dev/null echo " Password: $ADMIN_PASSWORD" echo " (Save these credentials securely!)" else - # Get credentials from .env - ADMIN_USERNAME=$(grep "ANOMSTACK_ADMIN_USERNAME=" ".env" 2>/dev/null | cut -d'=' -f2 | tr -d ' ' || echo "admin") - ADMIN_PASSWORD=$(grep "ANOMSTACK_ADMIN_PASSWORD=" ".env" 2>/dev/null | cut -d'=' -f2 | tr -d ' ') - echo "✅ Using admin credentials from .env file" + # Get credentials from environment file + ADMIN_USERNAME=$(grep "ANOMSTACK_ADMIN_USERNAME=" "$ENV_FILE" 2>/dev/null | cut -d'=' -f2 | tr -d ' ' || echo "admin") + ADMIN_PASSWORD=$(grep "ANOMSTACK_ADMIN_PASSWORD=" "$ENV_FILE" 2>/dev/null | cut -d'=' -f2 | tr -d ' ') + echo "✅ Using admin credentials from configuration" echo "🔑 Admin credentials:" echo " Username: $ADMIN_USERNAME" echo " Password: $ADMIN_PASSWORD" fi -# Add environment variables from .env file -if [[ -f ".env" ]]; then - echo "📁 Reading environment variables from .env..." +# Add environment variables from environment file +if [[ -f "$ENV_FILE" ]]; then + echo "📁 Reading environment variables from $ENV_FILE..." # Variables that should NOT be sent to Fly (local development only) skip_patterns=( @@ -150,7 +210,13 @@ if [[ -f ".env" ]]; then all_secrets+=("$var_name=$var_value") fi - done < ".env" + done < "$ENV_FILE" +fi + +# Clean up temporary file if we created one +if [[ -n "$TEMP_ENV_FILE" && -f "$TEMP_ENV_FILE" ]]; then + rm "$TEMP_ENV_FILE" + echo "🧹 Cleaned up temporary configuration file" fi # Set all secrets in one command to minimize releases @@ -184,6 +250,12 @@ echo "" echo "🎉 Deployment complete!" echo "📊 Public Dashboard: https://$APP_NAME.fly.dev/" echo "🔐 Admin Interface: https://$APP_NAME.fly.dev/dagster (login: $ADMIN_USERNAME/$ADMIN_PASSWORD)" + +# Show profile info if one was used +if [[ -n "$PROFILE" ]]; then + echo "🎯 Applied profile: $PROFILE" +fi + echo "" echo "Useful commands:" echo " fly logs -a $APP_NAME # View logs" @@ -191,7 +263,11 @@ echo " fly ssh console -a $APP_NAME # SSH into the app" echo " fly status -a $APP_NAME # Check status" echo " fly scale count 2 -a $APP_NAME # Scale to 2 instances" echo "" -echo "To set up alerting (optional):" +echo "Deploy with profiles:" +echo " ./scripts/deployment/deploy_fly.sh --profile demo # Deploy demo config" +echo " ./scripts/deployment/deploy_fly.sh --profile production # Deploy production config" +echo "" +echo "To set up alerting (if not configured in profile):" echo " fly secrets set ANOMSTACK_ALERT_EMAIL_FROM='your-email@domain.com' -a $APP_NAME" echo " fly secrets set ANOMSTACK_ALERT_EMAIL_TO='alerts@domain.com' -a $APP_NAME" echo " fly secrets set ANOMSTACK_ALERT_EMAIL_PASSWORD='your-app-password' -a $APP_NAME" \ No newline at end of file diff --git a/scripts/deployment/preview_fly_secrets.sh b/scripts/deployment/preview_fly_secrets.sh index 2fdd786c..3491053b 100755 --- a/scripts/deployment/preview_fly_secrets.sh +++ b/scripts/deployment/preview_fly_secrets.sh @@ -1,6 +1,24 @@ #!/bin/bash -echo "🔍 Previewing Fly.io secrets from .env file..." +echo "🔍 Previewing Fly.io secrets configuration..." + +# Parse command line arguments for profile support +PROFILE="" +while [[ $# -gt 0 ]]; do + case $1 in + --profile) + PROFILE="$2" + shift 2 + ;; + -p) + PROFILE="$2" + shift 2 + ;; + *) + shift + ;; + esac +done # Function to preview environment variables from .env file (matches deploy_fly.sh) preview_env_vars_from_file() { @@ -129,5 +147,46 @@ preview_env_vars_from_file() { echo " ./scripts/deployment/deploy_fly.sh [app-name]" } -# Run the preview -preview_env_vars_from_file ".env" \ No newline at end of file +# Handle deployment profile if specified (same logic as deploy_fly.sh) +if [[ -n "$PROFILE" ]]; then + PROFILE_FILE="profiles/${PROFILE}.env" + if [[ -f "$PROFILE_FILE" ]]; then + echo "🎯 Using deployment profile: $PROFILE" + echo "📄 Profile file: $PROFILE_FILE" + + # Create temporary merged .env file + TEMP_ENV_FILE=$(mktemp) + + # Start with existing .env if it exists + if [[ -f ".env" ]]; then + cat ".env" > "$TEMP_ENV_FILE" + echo "✅ Base configuration loaded from .env" + else + touch "$TEMP_ENV_FILE" + fi + + # Append profile configuration (profile values override .env values) + echo "" >> "$TEMP_ENV_FILE" # Add separator + echo "# Profile: $PROFILE (applied during deployment)" >> "$TEMP_ENV_FILE" + cat "$PROFILE_FILE" >> "$TEMP_ENV_FILE" + echo "✅ Profile configuration merged" + + # Use the merged file for preview + ENV_FILE="$TEMP_ENV_FILE" + + # Run the preview with merged configuration + preview_env_vars_from_file "$ENV_FILE" + + # Clean up + rm "$TEMP_ENV_FILE" + + else + echo "❌ Profile file not found: $PROFILE_FILE" + echo "Available profiles:" + ls -1 profiles/*.env 2>/dev/null | sed 's/profiles\///g' | sed 's/\.env//g' | sed 's/^/ - /' + exit 1 + fi +else + # Run the standard preview + preview_env_vars_from_file ".env" +fi \ No newline at end of file From 00dcf19d21e5259761256ee94e28095d4dba7495 Mon Sep 17 00:00:00 2001 From: andrewm4894 Date: Tue, 22 Jul 2025 16:25:43 +0100 Subject: [PATCH 2/3] feat: add Makefile targets for deployment profiles - Add fly-preview-demo, fly-preview-production, fly-preview-development - Add fly-deploy-demo, fly-deploy-production, fly-deploy-development - Enable easy deployment with: make fly-deploy-demo --- Makefile | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/Makefile b/Makefile index e51b53ab..f0a57a77 100644 --- a/Makefile +++ b/Makefile @@ -164,6 +164,8 @@ docker-prune: # ============================================================================= .PHONY: fly-validate fly-preview fly-deploy fly-status fly-logs fly-ssh +.PHONY: fly-preview-demo fly-preview-production fly-preview-development +.PHONY: fly-deploy-demo fly-deploy-production fly-deploy-development # validate fly.io configuration fly-validate: @@ -173,10 +175,34 @@ fly-validate: fly-preview: ./scripts/deployment/preview_fly_secrets.sh +# preview deployment with demo profile +fly-preview-demo: + ./scripts/deployment/preview_fly_secrets.sh --profile demo + +# preview deployment with production profile +fly-preview-production: + ./scripts/deployment/preview_fly_secrets.sh --profile production + +# preview deployment with development profile +fly-preview-development: + ./scripts/deployment/preview_fly_secrets.sh --profile development + # deploy to fly.io (reads .env file automatically) fly-deploy: ./scripts/deployment/deploy_fly.sh +# deploy to fly.io with demo profile (enables demo metric batches) +fly-deploy-demo: + ./scripts/deployment/deploy_fly.sh --profile demo + +# deploy to fly.io with production profile (production-ready settings) +fly-deploy-production: + ./scripts/deployment/deploy_fly.sh --profile production + +# deploy to fly.io with development profile (all examples enabled) +fly-deploy-development: + ./scripts/deployment/deploy_fly.sh --profile development + # check fly.io app status (requires app name as FLY_APP env var) fly-status: @if [ -z "$$FLY_APP" ]; then echo "Set FLY_APP environment variable"; exit 1; fi From 3e94e89f5aaad0ca1a754d29df52240d7ae2d1b9 Mon Sep 17 00:00:00 2001 From: andrewm4894 Date: Tue, 22 Jul 2025 16:26:24 +0100 Subject: [PATCH 3/3] docs: add comprehensive deployment profiles documentation - Add docs/docs/deployment/profiles.md with complete guide - Update fly.md with profiles reference - Add configuration management section to overview.md - Fix sidebar positions and typos in docker.md and gcp.md - Document profile composition, precedence, and workflows GIT_VALID_PII_OVERRIDE --- docs/docs/deployment/docker.md | 2 +- docs/docs/deployment/fly.md | 20 ++ docs/docs/deployment/gcp.md | 6 +- docs/docs/deployment/overview.md | 11 +- docs/docs/deployment/profiles.md | 392 +++++++++++++++++++++++++++++++ 5 files changed, 428 insertions(+), 3 deletions(-) create mode 100644 docs/docs/deployment/profiles.md diff --git a/docs/docs/deployment/docker.md b/docs/docs/deployment/docker.md index 1c7b252a..c11a976a 100644 --- a/docs/docs/deployment/docker.md +++ b/docs/docs/deployment/docker.md @@ -1,5 +1,5 @@ --- -sidebar_position: 1 +sidebar_position: 4 --- # Docker Deployment diff --git a/docs/docs/deployment/fly.md b/docs/docs/deployment/fly.md index 642fdcc5..e92aa95b 100644 --- a/docs/docs/deployment/fly.md +++ b/docs/docs/deployment/fly.md @@ -175,6 +175,26 @@ After deployment completes: - **📊 Dashboard**: `https://your-app-name.fly.dev/` - **🔐 Admin UI**: `https://your-app-name.fly.dev/dagster` (credentials shown after deploy) +## Deployment Profiles + +Anomstack supports **deployment profiles** that act like "Helm values files", allowing you to configure different environments (demo, production, development) without modifying the original metric batch examples. + +:::tip Learn More +📋 **[See the complete Deployment Profiles guide](profiles.md)** for detailed instructions on using profiles for environment-specific configurations. +::: + +**Quick examples:** +```bash +# Deploy demo instance (enables key examples) +make fly-deploy-demo + +# Deploy production instance (clean, focused) +make fly-deploy-production + +# Preview before deploying +make fly-preview-demo +``` + ## Manual Deployment For more control over the deployment process: diff --git a/docs/docs/deployment/gcp.md b/docs/docs/deployment/gcp.md index c8ae26c7..43d67caa 100644 --- a/docs/docs/deployment/gcp.md +++ b/docs/docs/deployment/gcp.md @@ -1,4 +1,8 @@ -# Goolge Cloud Platform +--- +sidebar_position: 5 +--- + +# Google Cloud Platform ## Deploying to a VM diff --git a/docs/docs/deployment/overview.md b/docs/docs/deployment/overview.md index 0973acbb..41a35797 100644 --- a/docs/docs/deployment/overview.md +++ b/docs/docs/deployment/overview.md @@ -136,6 +136,14 @@ Scalable options for production workloads. | **Kubernetes** | Multi-node, enterprise scale | | **Docker Swarm** | Simple orchestration | +### Configuration Management + +| Feature | Description | Compatibility | +|---------|-------------|---------------| +| **[Deployment Profiles](profiles.md)** | Environment-specific configurations (demo, production, dev) | Fly.io, Docker, Custom | +| **[Environment Variables](../configuration/environment-variables.md)** | Runtime configuration overrides | All platforms | +| **[Hot Reload](../configuration/hot-reload.md)** | Dynamic configuration updates | Docker, Local | + ## Architecture Patterns ### Pattern 1: All-in-One (Recommended for Getting Started) @@ -271,7 +279,8 @@ ANOMSTACK_ALERT_WEBHOOK_URL=https://api.company.com/alerts ### Deployment Guides - **[Docker Deployment](docker.md)** - Self-hosted with Docker Compose -- **[Fly.io Deployment](fly.md)** - Global edge deployment with managed infrastructure +- **[Fly.io Deployment](fly.md)** - Global edge deployment with managed infrastructure +- **[Deployment Profiles](profiles.md)** - Environment-specific configurations (demo, production, dev) - **[Google Cloud Deployment](gcp.md)** - GCP-native integration - **[Storage Optimization](storage-optimization.md)** - Optimize storage for large deployments diff --git a/docs/docs/deployment/profiles.md b/docs/docs/deployment/profiles.md new file mode 100644 index 00000000..c077caa6 --- /dev/null +++ b/docs/docs/deployment/profiles.md @@ -0,0 +1,392 @@ +--- +sidebar_position: 2 +--- + +# Deployment Profiles + +Deployment profiles act like "Helm values files" for Anomstack, allowing you to configure different environments without modifying the original metric batch examples. This approach gives you environment-specific overlays while keeping examples pristine as templates for users. + +## Overview + +Deployment profiles solve the common problem of wanting different configurations for different environments (demo, production, development) without duplicating code or modifying the original example metric batches. + +### Key Benefits + +✅ **Keep examples pristine** - Original examples remain as clean templates for users +✅ **Environment-specific configs** - Different settings for demo, dev, production +✅ **Source controlled** - All configurations are versioned and shareable +✅ **User-friendly** - Others can easily redeploy your exact configuration +✅ **Flexible layering** - Secrets in `.env`, configs in profiles +✅ **Hot-reloadable** - Changes can be applied without container restarts + +## How It Works + +Profiles use Anomstack's existing environment variable override system (`ANOMSTACK____`) to enable and configure metric batches at deployment time. + +### Configuration Layering + +1. **Your `.env` file**: API keys, secrets, personal settings +2. **Profile overlay**: Environment-specific metric batch configurations +3. **Merged deployment**: Combined configuration sent as deployment secrets + +```mermaid +graph LR + ENV[".env
🔐 Secrets & Personal"] --> MERGE[Deployment
Merger] + PROFILE["Profile
🎯 Environment Config"] --> MERGE + MERGE --> DEPLOY["🚀 Deployment
Combined Config"] +``` + +## Available Profiles + +### Demo Profile (`profiles/demo.env`) + +**Purpose**: Configuration for public demo instances +**Use Case**: Showcase Anomstack capabilities with real examples + +```bash +# Enable examples for demo visibility +ANOMSTACK_IGNORE_EXAMPLES=no + +# Enable netdata infrastructure monitoring demo +ANOMSTACK__NETDATA__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__LLMALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__PLOT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__NETDATA__ALERT_ALWAYS=True +ANOMSTACK__NETDATA__ALERT_METHODS=email + +# Enable weather API integration demo +ANOMSTACK__WEATHER__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__WEATHER__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__WEATHER__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__WEATHER__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__WEATHER__INGEST_CRON_SCHEDULE=*/30 * * * * + +# Enable hackernews viral detection demo +ANOMSTACK__HN_TOP_STORIES_SCORES__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__HN_TOP_STORIES_SCORES__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__HN_TOP_STORIES_SCORES__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__HN_TOP_STORIES_SCORES__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING + +# Demo-specific global settings +ANOMSTACK_ALERT_EMAIL_FROM=demo@anomstack.com +ANOMSTACK_TABLE_KEY=demo_metrics +ANOMSTACK_LLM_PLATFORM=openai +ANOMSTACK_OPENAI_MODEL=gpt-4o-mini +``` + +### Production Profile (`profiles/production.env`) + +**Purpose**: Production-ready deployment template +**Use Case**: Conservative, focused production deployments + +```bash +# Disable examples in production for clean deployment +ANOMSTACK_IGNORE_EXAMPLES=yes + +# Production-grade database configuration +ANOMSTACK_TABLE_KEY=production.metrics +ANOMSTACK_MODEL_PATH=gs://your-company-models/anomstack/ + +# Reliable email alerting +ANOMSTACK_ALERT_EMAIL_FROM=anomstack-alerts@yourcompany.com +ANOMSTACK_ALERT_EMAIL_TO=ops-team@yourcompany.com,data-team@yourcompany.com + +# Conservative alert settings +ANOMSTACK_ALERT_THRESHOLD=0.8 +ANOMSTACK_ALERT_METRIC_TIMESTAMP_MAX_DAYS_AGO=7 + +# Production LLM settings +ANOMSTACK_LLM_PLATFORM=anthropic +ANOMSTACK_ANTHROPIC_MODEL=claude-3-sonnet-20240229 + +# Production Dagster settings +ANOMSTACK_DAGSTER_OVERALL_CONCURRENCY_LIMIT=3 +ANOMSTACK_MAX_RUNTIME_SECONDS_TAG=7200 +``` + +### Development Profile (`profiles/development.env`) + +**Purpose**: Development and staging environments +**Use Case**: Testing with all examples enabled and sensitive settings + +```bash +# Enable all examples for development testing +ANOMSTACK_IGNORE_EXAMPLES=no + +# Development database configuration +ANOMSTACK_TABLE_KEY=dev_metrics +ANOMSTACK_MODEL_PATH=local://./tmp/models + +# Development-friendly alert settings +ANOMSTACK_ALERT_EMAIL_FROM=dev-anomstack@yourcompany.com +ANOMSTACK_ALERT_EMAIL_TO=dev-team@yourcompany.com +ANOMSTACK_ALERT_THRESHOLD=0.6 +ANOMSTACK_ALERT_ALWAYS=True + +# Enable hot reload features for development +ANOMSTACK_AUTO_CONFIG_RELOAD=true +ANOMSTACK_CONFIG_WATCHER=true +ANOMSTACK_CONFIG_WATCHER_INTERVAL=30 + +# Enable key examples for development testing +ANOMSTACK__PYTHON_INGEST_SIMPLE__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__PYTHON_INGEST_SIMPLE__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__PYTHON_INGEST_SIMPLE__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__PYTHON_INGEST_SIMPLE__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING + +ANOMSTACK__EXAMPLE_SIMPLE__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__EXAMPLE_SIMPLE__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__EXAMPLE_SIMPLE__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__EXAMPLE_SIMPLE__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +``` + +## Using Profiles + +### Deployment Commands + +**With Makefile (recommended):** +```bash +# Deploy demo instance (enables key examples) +make fly-deploy-demo + +# Deploy production instance (clean, focused) +make fly-deploy-production + +# Deploy development instance (all examples enabled) +make fly-deploy-development +``` + +**With deployment script:** +```bash +# Deploy with specific profile +./scripts/deployment/deploy_fly.sh --profile demo my-demo-app + +# Deploy with default app name +./scripts/deployment/deploy_fly.sh --profile production +``` + +### Preview Commands + +Always preview your deployment configuration before deploying: + +```bash +# Preview demo deployment +make fly-preview-demo + +# Preview production deployment +make fly-preview-production + +# Preview with script +./scripts/deployment/preview_fly_secrets.sh --profile demo +``` + +### Configuration Precedence + +Profiles override your personal `.env` settings while preserving secrets: + +**Example precedence:** +```bash +# Your .env file (base layer - secrets preserved) +ANOMSTACK_OPENAI_KEY=sk-your-secret-key # ✅ Preserved +ANOMSTACK_ALERT_EMAIL_FROM=you@company.com # ✅ Preserved +ANOMSTACK_TABLE_KEY=my_personal_metrics # ❌ Will be overridden + +# Profile layer (demo.env - takes precedence) +ANOMSTACK_TABLE_KEY=demo_metrics # ✅ Wins! +ANOMSTACK_IGNORE_EXAMPLES=no # ✅ Added +ANOMSTACK__NETDATA__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING # ✅ Added + +# Final deployment result +ANOMSTACK_OPENAI_KEY=sk-your-secret-key # From .env +ANOMSTACK_ALERT_EMAIL_FROM=you@company.com # From .env +ANOMSTACK_TABLE_KEY=demo_metrics # From profile +ANOMSTACK_IGNORE_EXAMPLES=no # From profile +ANOMSTACK__NETDATA__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING # From profile +``` + +## Creating Custom Profiles + +### Company-Specific Profile + +Create profiles tailored to your organization: + +```bash +# Create custom profile +cat > profiles/acme-corp.env << EOF +# ACME Corp Production Anomstack Profile +ANOMSTACK_IGNORE_EXAMPLES=yes +ANOMSTACK_TABLE_KEY=acme.production.metrics +ANOMSTACK_MODEL_PATH=gs://acme-ml-models/anomstack/ + +# Enable ACME-specific metric batches +ANOMSTACK__SALES_METRICS__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__SALES_METRICS__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__SALES_METRICS__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__SALES_METRICS__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING + +ANOMSTACK__USER_METRICS__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__USER_METRICS__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__USER_METRICS__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__USER_METRICS__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING + +# ACME alert configuration +ANOMSTACK_ALERT_EMAIL_FROM=anomstack@acme-corp.com +ANOMSTACK_ALERT_EMAIL_TO=data-team@acme-corp.com,ops@acme-corp.com +ANOMSTACK_SLACK_CHANNEL=#data-alerts +EOF + +# Deploy with custom profile +./scripts/deployment/deploy_fly.sh --profile acme-corp acme-anomstack-prod +``` + +### Multi-Environment Setup + +Use different profiles for different environments: + +```bash +# Development environment +./scripts/deployment/deploy_fly.sh --profile development acme-anomstack-dev + +# Staging environment (using production profile as base) +./scripts/deployment/deploy_fly.sh --profile production acme-anomstack-staging + +# Production environment +./scripts/deployment/deploy_fly.sh --profile acme-corp acme-anomstack-prod + +# Demo environment for stakeholders +./scripts/deployment/deploy_fly.sh --profile demo acme-anomstack-demo +``` + +## Environment Variable Format + +Profiles use Anomstack's metric batch override system: + +### Format + +```bash +ANOMSTACK____= +``` + +**Rules:** +- ``: Uppercase metric batch name with dashes replaced by underscores +- ``: Uppercase parameter name +- ``: The configuration value + +### Common Parameters + +**Schedule Controls:** +```bash +ANOMSTACK__BATCH_NAME__INGEST_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__BATCH_NAME__TRAIN_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__BATCH_NAME__SCORE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__BATCH_NAME__ALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__BATCH_NAME__CHANGE_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__BATCH_NAME__LLMALERT_DEFAULT_SCHEDULE_STATUS=RUNNING +ANOMSTACK__BATCH_NAME__PLOT_DEFAULT_SCHEDULE_STATUS=RUNNING +``` + +**Alert Configuration:** +```bash +ANOMSTACK__BATCH_NAME__ALERT_METHODS=email,slack +ANOMSTACK__BATCH_NAME__ALERT_THRESHOLD=0.7 +ANOMSTACK__BATCH_NAME__ALERT_ALWAYS=True +``` + +**Scheduling:** +```bash +ANOMSTACK__BATCH_NAME__INGEST_CRON_SCHEDULE="*/15 * * * *" +ANOMSTACK__BATCH_NAME__TRAIN_CRON_SCHEDULE="*/60 * * * *" +``` + +## Profile Composition Workflows + +### For Demo Creators + +1. **Develop your demo configuration**: + ```bash + # Test locally first + cp profiles/demo.env .env + # Add your API keys to .env + make local + ``` + +2. **Deploy your demo**: + ```bash + make fly-deploy-demo my-awesome-demo + ``` + +3. **Share with others**: + ```bash + # Others can recreate your exact demo + git clone your-repo + cp .example.env .env # They add their API keys + make fly-deploy-demo their-demo-name + ``` + +### For Production Teams + +1. **Create company profile**: + ```bash + vim profiles/company-prod.env + # Configure your production-specific settings + ``` + +2. **Test in staging**: + ```bash + ./scripts/deployment/deploy_fly.sh --profile company-prod staging-app + ``` + +3. **Deploy to production**: + ```bash + ./scripts/deployment/deploy_fly.sh --profile company-prod production-app + ``` + +### For Multi-Tenant SaaS + +```bash +# Create tenant-specific profiles +vim profiles/tenant-a.env +vim profiles/tenant-b.env + +# Deploy separate instances +./scripts/deployment/deploy_fly.sh --profile tenant-a tenant-a-anomstack +./scripts/deployment/deploy_fly.sh --profile tenant-b tenant-b-anomstack +``` + +## Best Practices + +### Profile Organization + +- **Keep profiles focused**: One environment or use case per profile +- **Use descriptive names**: `company-production.env` vs `prod.env` +- **Document purpose**: Add comments explaining profile intent +- **Version control**: Always commit profiles (they don't contain secrets) + +### Security + +- **Secrets in `.env`**: Never put API keys or passwords in profiles +- **Public profiles**: Profiles are source-controlled and visible +- **Sensitive overrides**: Use deployment-time environment variables for sensitive config + +### Testing + +- **Always preview**: Use `make fly-preview-` before deploying +- **Test in staging**: Use profiles in non-production first +- **Validate configurations**: Ensure all required variables are set + +### Documentation + +- **Document your profiles**: Add README files explaining custom profiles +- **Share examples**: Provide example `.env` files for your profiles +- **Update regularly**: Keep profiles in sync with new features + +## Related Documentation + +- **[Environment Variables](../configuration/environment-variables.md)** - Complete list of available environment variables +- **[Metrics Configuration](../configuration/metrics.md)** - How to configure metric batches +- **[Fly.io Deployment](fly.md)** - Fly.io-specific deployment details +- **[Hot Reload](../configuration/hot-reload.md)** - Dynamic configuration updates \ No newline at end of file