Skip to content

hl-782: Add MMGIS Copilot backed by Azure AI Agent Service #783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
midatm1234 wants to merge 22 commits into
base: development
Choose a base branch
from
Open

hl-782: Add MMGIS Copilot backed by Azure AI Agent Service #783

midatm1234 wants to merge 22 commits into from

Conversation

@midatm1234
Copy link

@midatm1234 midatm1234 commented Oct 27, 2025

Purpose

  • Introduce the MMGIS Copilot experience that connects to Azure AI Agent Service for mission-aware assistance.
  • Wire backend, tooling registry, and deployment assets so Copilot can issue MMGIS actions across environments.

Proposed Changes

  • [ADD] AgentChat floating panel with history/undo support, registry-driven tool metadata, Azure provider integration, and validation/test scripts.
  • [CHANGE] Docker/Docker Compose, sample envs, and docs to expose Azure authentication flow and mount CLI token caches.
  • [FIX] Remove outdated Azure auth guidance from sample.env and tighten layer-tool dispatch logic uncovered while enabling Copilot.

Issues

Testing

  • PManual: launched docker-compose up with Azure credentials to verify Copilot start-up, layer toggles, and undo flow.
  • Scripts: scripts/test-provider.js, scripts/test-ajv.js, and scripts/test-e2e-agent.js (pass).

edwin-jpl and others added 17 commits September 26, 2025 08:45
@edwin-jpl
feat(agent): add minimal chatbox tool + backend with Azure tool-calli…
dd1348a 
…ng and validation

WHY
- Deliver MVP loop for MMGIS cyber agent: map control via plain English with
    only four actions (list layers, toggle, opacity, zoom). This keeps the (code) surface
    area tiny, reduces risk, and enables fast demos on the built-in "Test" mission.
- Use same-origin `/api/agent` and MMGIS' auto-discovery for tools/backends to avoid
    core edits and CORS complexity. This aligns with MMGIS' plugin guidance and keeps
    rollback trivial.
- Add a rule-based fallback so the UI is demoable even if the LLM is misconfigured,
    de-risking first-run experience while we wire credentials.

WHAT
- Frontend Tool: `src/essence/Tools/AgentChat/`
    - Chat panel (input + send + transcript).
    - Executes validated actions via the client JS API (list/toggle/opacity/zoom).
    - Same-origin call to `/api/agent`; uses ROOT_PATH for robustness.
- Backend Plugin: `API/Backend/Agent/`
    - `setup.js` registers `POST /api/agent` during `onceInit`.
    - `routes/agent.js`:
    - Validates tool calls (name + args), clamps opacity, checks lon/lat ranges,
        enforces zoom bounds, and summarizes dropped/invalid calls.
    - Includes a tiny rules mapper for "list", "turn on Sample_Points", "opacity 0.7",
        and "zoom to lon,lat [zoom]" to prove the loop offline.
    - `provider.js`:
    - Thin Azure OpenAI caller using function-calling/tools with JSON Schemas
        for the four actions; extracts `tool_calls` and returns `{tool,args}` tuples.
- Tool manifest: `src/essence/Tools/AgentChat/config.json` (autoloaded; icon, paths, copy).

FOLLOW-UPS
- If any `mmgisAPI.*` helpers differ in your MMGIS build, swap to the documented
    layer/visibility/zoom helpers from the JavaScript API page.
@edwin-jpl
feat(ChatAgent): Use OpenAI Azure SDK, single-source UX, add undo action
abf5de3 
WHY
- Close acceptance gaps and harden UX: human display names with visibility,
  a single source of truth for success messages, strict validation, and an
  observable provider path with graceful fallback.
- Reduce operational variance by using the Azure SDK instead of raw REST API.
- Keep the surface area small and maintainable.

WHAT
- Provider (`provider.js`): switch to AzureOpenAI SDK chat completions
  with tools; set parallel_tool_calls=false, and a 10s timeout; include
  `source` and `debug` in responses for traceability.
- Server (`agent.js`): validate actions strictly, drop unknown tools with
  reasons, and return a neutral "Planned:" message instead of claiming execution.
- UI (`AgentChatTool.js`): build a layer catalog of display names with
  visibility; resolve names to internal ids before execution; perform actions
  once; print one authoritative "Performed:" line; add Undo for
  toggle/opacity/zoom; add input loading state, message length guard, and
  focus restore.

TESTS
- Add `test_provider.js` to exercise the provider in isolation. It accepts
  a prompt, runs through the SDK with our tool schemas and timeout, and
  prints a compact request/response summary (choices, tool calls, reason).
  When creds are missing it cleanly reports the fallback path. Useful for
  local smoke tests and CI diagnostics without spinning up the full UI.

OUTCOMES
- Deterministic behavior with transparent fallback and clear transcripts.
- Human-friendly layer lists that match what users see on the map.
- Demos work online (provider) and offline (fallback) with minimal code.
@edwin-jpl
feat(agent): registry‑driven tools, AJV validation, generic dispatch
87ed7a6 
Motivation
Replace brittle, duplicated tool wiring with a single source of truth.
Align with DRY principles so planning, validation, and execution stay in sync.

- Add generator (scripts/generate-tool-registry.js) to read per‑tool JSON
  specs in API/Backend/Agent/tools/*.json and emit API/Backend/Agent/tool-registry.json.
- Server: load registry on boot, compile AJV validators, expose
  GET /api/agent/tools; schema‑driven validation with actionable errors.
- Provider: define model tools from registry; use modelParameters
  (no top‑level oneOf) while server uses strict parameters for AJV.
- Client (AgentChat): replace switches with registry‑backed dispatcher:
  - adapter:"mmgisAPI" maps args (displayName→id) to window.mmgisAPI.
  - adapter:"custom" handles list_layers/zoom/opacity with undo.
- Add tool specs: list_layers, toggle_layer, set_layer_opacity, zoom_to.
- package.json: add gen:tools and ajv dependency.
- configure/public/toolConfigs.json: add AgentChat tool config.
@edwin-jpl
feat(agent): enforce DRY with registry and renderers
d6092da 
Motivation
- Remove duplicated tool logic across server and client so planning,
  validation, and execution remain consistent. Keep complex I/O
  (e.g., web search) on the server; keep the client a thin, generic
  renderer.

What changed
- Provider exports OpenAI/Azure tools solely from the registry
  (name/description/parameters via modelParameters) and parses
  tool_calls into {tool,args} without prior knowledge of names.
- Server planning/validation is registry-driven with Ajv; added a
  server /api/agent/exec entrypoint for server tools (initial
  web_search_product) returning {summary,links,citations}.
- Registry/generator adds describe_mmgis, describe_layer, and
  web_search_product; emits uiProfiles for renderer mapping.
- Client replaces the if/else ladder with a renderer map and dispatch
  by exec.adapter + uiProfiles[type].kind; undo stores generic
  descriptors instead of tool-name branches.
- Tests add schema checks and basic provider/exec smoke coverage.

Design principles
- Single source of truth (registry) for contracts and UI hints.
- Adapters, not switches (mmgisAPI/openapi/custom).
- Validation-first (Ajv, helpful errors, clamping where safe).
- Deterministic generator and provider export for stable diffs.

Notes
- Fallback planner no longer routes by keywords.
- Web search currently uses Bing REST behind a stable interface; next
  step is Azure Agents grounding with Bing Search.

BREAKING CHANGES
- None; internal code must not rely on tool-name conditionals.
@edwin-jpl
feat(agent): adopt Azure AI Agent Service; unify provider & routes
0320151 
Replaces the direct Azure OpenAI chat completions client with Azure
AI Agents SDK (@azure/ai-agents) for all agent planning and general
Q&A; refactor provider to plan via Agents; simplify /api/agent route;
update registry and remove legacy tools.

The AI agent now leverages built-in Bing Grounding for general Q&A,
which provides grounded replies and source citations directly from the
service, eliminating the need for custom server-side execution.

Key changes:
- Introduces a new `azureService.js` module to encapsulate all
  `AgentsClient` interactions (thread creation, runs, and cleanup).
- Removes the now-redundant `/api/agent/exec` endpoint and its
  brittle, manual web search implementation.
- Simplifies the `provider.js` to build a single prompt and parse a
  structured JSON response, offloading all planning logic.

This change aligns our architecture with the recommended pattern for
Azure AI Foundry, significantly reduces code complexity, and improves
both maintainability and future extensibility.

BREAKING CHANGE: deprecated tools removed and provider now requires
PROJECT_ENDPOINT and AZURE_AI_FOUNDRY_AGENT_ID.
@edwin-jpl
feat(ui): rebuild AgentChat as a dependency-free floating panel
6dda699 
Rewrites `src/essence/Tools/AgentChat/AgentChatTool.js` as a draggable,
resizable, minimizable overlay; persists history, shows citations, and exposes
a traces/debug section. The previous jQuery-based implementation
has been removed. The new component is built with plain JavaScript.

Harden renderers.js: strict payload validation; consistent output
for links/summary/citations; safer layer lookup/zoom/opacity.
for git rebase upstream/cyber_agent_fas
bbff28b
chore: mount Azure CLI cache into mmgis container
9bf39ed
chore: install Azure CLI and mount host tokens
6af3f56
enable Azure CLI authentication in mmgis Docker container'
5b7c99b
Azure authentication information has been removed in sample.env
1daee53
Add agent tool specs for advanced layer analytics
a2ed47a
Improve agent backend for fuzzy layer reasoning
82f3fe7
Expand AgentChat UI for intelligent layer actions
d681520
chore: document Azure AI agent setup
126521c
feat(agentchat): add trash clear control and list layers with bullets
88705b7
feat(agentchat): delegate layer similarity to azure agent
9a30b38
tariqksoliman added a commit that referenced this pull request Oct 27, 2025
@tariqksoliman
#783 Fix conflict, minor style change
cc78aa8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@midatm1234 @edwin-jpl