Skip to content

Commit 464d9a3

Browse files
ankurgitksankurgitks
committed
feat: Add comprehensive alpha-release-testing framework for AI observability
This PR introduces a complete testing framework for validating AI observability features in Splunk O11y, including multiple instrumentation patterns and evaluation metrics testing. Key Changes: - Add alpha-release-testing framework with 5 test applications - Add retail_shop_langchain_app.py: 3-agent retail system with unified traces - Add direct_azure_openai_app.py: Manual GenAI instrumentation without frameworks - Add comprehensive documentation for zero-code vs manual instrumentation - Add Docker and Kubernetes deployment configurations - Add environment variables reference (30+ configuration options) - Fix LangchainInstrumentor entry point case sensitivity issue - Remove redundant azure-ai-validation directory - Replace all hardcoded API keys and tokens with placeholders - Update pyproject.toml to exclude examples from linting - Update requirements to use PyPI version of traceloop translator Applications: 1. retail_shop_langchain_app.py - LangChain auto-instrumentation 2. langchain_evaluation_app.py - Deterministic evaluation testing 3. langgraph_travel_planner_app.py - Complex workflow orchestration 4. direct_azure_openai_app.py - Manual GenAI instrumentation 5. traceloop_travel_planner_app.py - Framework-agnostic instrumentation Validation: - All apps tested on RC0 environment - Evaluation metrics visible on all agents - Unified trace structure confirmed - Token usage metrics accurate - Documentation matches customer requirements Fixes: QSE-4732 Signed-off-by: ankurs <[email protected]>
1 parent de8987b commit 464d9a3

32 files changed

+7452
-2
lines changed

‎instrumentation-genai/opentelemetry-instrumentation-langchain/examples/alpha-release-testing/.gitignore‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Alpha Release Testing - Git Ignore
2+
3+
# Environment files with credentials
4+
.env
5+
.env.*
6+
!.env.*.template
7+
config/.env.*
8+
!config/.env.*.template
9+
10+
# Python
11+
__pycache__/
12+
*.py[cod]
13+
*$py.class
14+
*.so
15+
.Python
16+
build/
17+
develop-eggs/
18+
dist/
19+
downloads/
20+
eggs/
21+
.eggs/
22+
lib/
23+
lib64/
24+
parts/
25+
sdist/
26+
var/
27+
wheels/
28+
*.egg-info/
29+
.installed.cfg
30+
*.egg
31+
32+
# Virtual Environment
33+
.venv/
34+
venv/
35+
ENV/
36+
env/
37+
38+
# Testing
39+
.pytest_cache/
40+
.coverage
41+
htmlcov/
42+
*.cover
43+
.hypothesis/
44+
.tox/
45+
logs/*.log
46+
logs/*.html
47+
logs/*.xml
48+
logs/*.json
49+
!logs/.gitkeep
50+
51+
# IDE
52+
.vscode/
53+
.idea/
54+
*.swp
55+
*.swo
56+
*~
57+
.DS_Store
58+
59+
# Playwright
60+
test-results/
61+
playwright-report/
62+
playwright/.cache/
63+
64+
# Temporary files
65+
*.tmp
66+
*.bak
67+
*.swp
68+
temp/
69+
tmp/
70+
71+
# Test data
72+
test_data/
73+
*.db
74+
*.sqlite
75+
76+
# Screenshots (UI tests)
77+
screenshots/
78+
*.png
79+
!docs/*.png
80+
81+
# Credentials and secrets
82+
secrets/
83+
*.pem
84+
*.key
85+
*.crt
86+
credentials.json

‎instrumentation-genai/opentelemetry-instrumentation-langchain/examples/alpha-release-testing/README.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# Alpha Release Testing
2+
3+
Manual testing framework for validating Alpha release AI observability features against customer documentation.
4+
5+
## 📁 Structure
6+
7+
```
8+
alpha-release-testing/
9+
├── config/
10+
│ └── .env # Single configuration file (edit this)
11+
├── tests/apps/ # Test applications
12+
│ ├── retail_shop_langchain_app.py # NEW: Retail multi-agent (unified traces)
13+
│ ├── langchain_evaluation_app.py # LangChain multi-agent (6 scenarios)
14+
│ ├── langgraph_travel_planner_app.py # LangGraph workflow (5 agents)
15+
│ ├── direct_azure_openai_app.py # Manual GenAI instrumentation
16+
│ └── traceloop_travel_planner_app.py # Traceloop translator
17+
├── docs/
18+
│ ├── ALPHA_RELEASE_TEST_PLAN.md # Test plan with all use cases
19+
│ └── TEST_EXECUTION_CHECKLIST.md # Execution tracking
20+
└── README.md # This file
21+
```
22+
23+
## 🎯 Purpose
24+
25+
Validate customer documentation use cases:
26+
- Instrument AI Applications (zero-code & code-based)
27+
- LangChain/LangGraph instrumentation
28+
- Traceloop SDK integration
29+
- Configuration settings
30+
- Splunk APM UI verification
31+
32+
## 🚀 Quick Start
33+
34+
### One-Time Setup
35+
36+
```bash
37+
cd alpha-release-testing
38+
39+
# Run setup script (one time only)
40+
./setup.sh
41+
42+
# Edit config/.env and verify your OPENAI_API_KEY
43+
vim config/.env
44+
```
45+
46+
### Run Tests (Automated)
47+
48+
```bash
49+
# Run all tests once (includes both zero-code and manual modes)
50+
./run_tests.sh
51+
52+
# Run only LangChain test
53+
./run_tests.sh langchain
54+
55+
# Run LangGraph test (both zero-code and manual modes)
56+
./run_tests.sh langgraph
57+
58+
# Run LangGraph with zero-code instrumentation only
59+
./run_tests.sh langgraph_zerocode
60+
61+
# Run LangGraph with manual instrumentation only
62+
./run_tests.sh langgraph_manual
63+
64+
# Run all tests continuously every 30 seconds
65+
./run_tests.sh loop_30
66+
67+
# Run only LangChain test every 60 seconds
68+
./run_tests.sh langchain loop_60
69+
70+
# Run only LangGraph test every 120 seconds
71+
./run_tests.sh langgraph loop_120
72+
```
73+
74+
The script automatically:
75+
- Activates virtual environment
76+
- Loads environment variables (with proper export)
77+
- Runs selected test application(s)
78+
- **LangGraph runs in BOTH modes**: Zero-code (opentelemetry-instrument) and Manual (hardcoded)
79+
- Shows summary of results
80+
- **Loop mode**: Runs continuously at specified intervals (Press Ctrl+C to stop)
81+
82+
---
83+
84+
## 📝 Manual Setup (Alternative)
85+
86+
If you prefer manual setup:
87+
88+
### 1. Install Dependencies
89+
90+
```bash
91+
cd alpha-release-testing
92+
93+
# Create virtual environment
94+
uv venv .venv-langchain
95+
source .venv-langchain/bin/activate
96+
97+
# Install pip
98+
uv pip install pip
99+
100+
# Install local Splunk packages
101+
pip install -e ../../../../util/opentelemetry-util-genai --no-deps && \
102+
pip install -e ../../../../util/opentelemetry-util-genai-emitters-splunk --no-deps && \
103+
pip install -e ../../../../util/opentelemetry-util-genai-evals --no-deps && \
104+
pip install -e ../../../../util/opentelemetry-util-genai-evals-deepeval && \
105+
pip install -e ../../../../instrumentation-genai/opentelemetry-instrumentation-langchain/
106+
```
107+
108+
### 2. Configure Environment
109+
110+
```bash
111+
# Edit the single .env file
112+
vim config/.env # Update OPENAI_API_KEY, SPLUNK_REALM, SPLUNK_ACCESS_TOKEN
113+
114+
# Export environment variables (important!)
115+
set -a
116+
source config/.env
117+
set +a
118+
```
119+
120+
### 3. Run Tests Manually
121+
122+
```bash
123+
cd tests/apps
124+
125+
# LangChain evaluation (6 scenarios)
126+
python langchain_evaluation_app.py
127+
128+
# LangGraph travel planner - Manual instrumentation (hardcoded)
129+
python langgraph_travel_planner_app.py
130+
131+
# LangGraph travel planner - Zero-code instrumentation
132+
opentelemetry-instrument python langgraph_travel_planner_app.py
133+
```
134+
135+
## 📊 Verify in Splunk APM
136+
137+
1. Navigate to Splunk APM (check your `SPLUNK_REALM` in config/.env)
138+
- rc0: https://app.rc0.signalfx.com
139+
- us1: https://app.us1.signalfx.com
140+
- lab0: https://app.lab0.signalfx.com
141+
2. Go to **APM → Traces**
142+
3. Search for service: `sf_service:alpha-release-test`
143+
4. Verify:
144+
- Agent names appear correctly
145+
- Evaluation metrics visible
146+
- Token usage tracked
147+
- Trace hierarchy correct
148+
149+
## 📚 Documentation
150+
151+
- **Test Plan**: `docs/ALPHA_RELEASE_TEST_PLAN.md` - All test cases and use cases
152+
- **Checklist**: `docs/TEST_EXECUTION_CHECKLIST.md` - Track execution progress
153+
- **Test Apps**: `tests/apps/README.md` - Detailed app documentation
154+
155+
## 🔧 Troubleshooting
156+
157+
**Environment variables not loaded:**
158+
```bash
159+
# Verify environment is loaded
160+
echo $OPENAI_API_KEY
161+
echo $OTEL_SERVICE_NAME
162+
163+
# Reload if needed
164+
source config/.env
165+
```
166+
167+
**Import errors:**
168+
```bash
169+
# Verify virtual environment is active
170+
which python # Should show .venv-langchain/bin/python
171+
172+
# Reinstall packages if needed
173+
pip install -e ../../../../instrumentation-genai/opentelemetry-instrumentation-langchain/
174+
```
175+
176+
**No telemetry in Splunk:**
177+
- Check OTEL Collector is running: `curl http://localhost:4317`
178+
- Verify `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env`
179+
- Check service name matches in Splunk APM
180+
181+
---
182+
183+
**Status**: Ready for manual testing
184+
**Configuration**: Single `config/.env` file (no templates)
185+
**Last Updated**: November 12, 2025

‎instrumentation-genai/opentelemetry-instrumentation-langchain/examples/alpha-release-testing/config/.env.lab0.template‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Alpha Release Testing - lab0 Environment Configuration
2+
# Copy this file to .env.lab0 and configure for your environment
3+
4+
OPENAI_API_KEY=your-openai-api-key-here
5+
6+
# =============================================================================
7+
# Splunk Observability Cloud Configuration - lab0
8+
# =============================================================================
9+
SPLUNK_REALM=lab0
10+
SPLUNK_ACCESS_TOKEN=your-lab0-access-token-here
11+
SPLUNK_HEC_TOKEN=your-lab0-hec-token-here
12+
SPLUNK_HEC_URL=https://bits.splunk.com:8088/services/collector/event
13+
SPLUNK_COLLECTD_DIR=/usr/local/opt/collectd
14+
15+
# =============================================================================
16+
# OpenTelemetry Core Configuration
17+
# =============================================================================
18+
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
19+
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
20+
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA
21+
OTEL_LOGS_EXPORTER=otlp
22+
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
23+
24+
# =============================================================================
25+
# Service Configuration
26+
# =============================================================================
27+
OTEL_SERVICE_NAME=alpha-release-test
28+
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=ai-test-val,test.phase=alpha,realm=lab0
29+
30+
# =============================================================================
31+
# GenAI Instrumentation Configuration
32+
# =============================================================================
33+
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
34+
OTEL_INSTRUMENTATION_GENAI_EMITTERS=span_metric_event,splunk
35+
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
36+
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE=SPAN_AND_EVENT
37+
OTEL_INSTRUMENTATION_GENAI_EVALS_RESULTS_AGGREGATION=true
38+
OTEL_INSTRUMENTATION_GENAI_DEBUG=false
39+
40+
# =============================================================================
41+
# Evaluation Configuration
42+
# =============================================================================
43+
OTEL_INSTRUMENTATION_GENAI_EVALS_EVALUATORS="deepeval(LLMInvocation(bias,toxicity,hallucination,relevance,sentiment))"
44+
OTEL_INSTRUMENTATION_GENAI_EVALUATION_SAMPLE_RATE=1.0
45+
OTEL_GENAI_EVAL_DEBUG_SKIPS=false
46+
OTEL_GENAI_EVAL_DEBUG_EACH=false
47+
48+
# =============================================================================
49+
# DeepEval Configuration
50+
# =============================================================================
51+
DEEPEVAL_FILE_SYSTEM=READ_ONLY
52+
DEEPEVAL_TELEMETRY_OPT_OUT=YES
53+
54+
# =============================================================================
55+
# Azure OpenAI Configuration
56+
# =============================================================================
57+
AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com
58+
AZURE_OPENAI_API_KEY=your-azure-openai-api-key-here
59+
AZURE_OPENAI_DEPLOYMENT=gpt-4
60+
AZURE_OPENAI_API_VERSION=2024-08-01-preview
61+
62+
# =============================================================================
63+
# LangChain Instrumentation
64+
# =============================================================================
65+
OTEL_INSTRUMENTATION_LANGCHAIN_CAPTURE_MESSAGE_CONTENT=true

0 commit comments

Comments
 (0)