Skip to content

Commit 5bc99c2

Browse files
sjrlanakin87
sjrl
and
anakin87
authored
feat: Add Human in the Loop confirmation strategies for tool execution in Agent (#369)
* Initial commit * Refactoring * Add more docstrings * Add agent example * doc strings * Formatting * Linting and mypy * PR comments Some renaming and example how I'd like the interface to look Alternative implementation formatting Continue work on alternative Fixes Add more examples Change name of file * Add missing header * Formatting * PR comments * PR comments * docstrings and refactoring * Cleaning up and moving some things to base classes instead of protocols * Start adding pydoc files * Start updating readme * Add new example using multi-agent architecture and refactoring * Refactoring * More refactoring * Make UIs threadsafe * Update docs * formatting * Formatting * linting * Update readme * Update pydoc * Fix docs * Fix docs * Add update_after_confirmation function to allow better control of updating a Policy's state * PR comments * update pydocs * PR comments and starting to add tests * formatting * PR comments and added new execute param to ToolExecutionDecision * Add docstrings, update how modify parameters works * Update parameter modification prompt * Make parameter modification more robust and user friendly * Add more docstrings * Add more tests * Update tests * Remove file * Refactor to make the policies, user interfaces and strategies to directly take in tool name and description instead of a tool instance. Should be better for exchanging information over a RestAPI * feat: Update HiTL confirmation strategies to work with Breakpoints (#372) * Exploring supporting hitl using breakpoints * Add example script * Update example so using only break points to stop and start execution works * Keep working on the examples * Updating example * Making progress, the confirm, and modify both work now. The rejection option still has a problem. * Get example to work * Work on some of the TODOs * Add tool_id to ToolExecutionDecision * Use the serialized chat messages instead of just the tool calls * Minor formatting update * Refactoring * Refactor to move confirmation strategy logic to Agent instead of ToolInvoker * More refactoring and prep for getting BreakpointConfirmationStrategy to work * Add missing run_async to Agent and remove unused test file * Add missing import * First version of HiTL + Breakpoints is working! * Some refactoring and adding a TODO * Refactoring example script to be a bit more robust * Fixed some bugs * Some cleanup * Rename file * Fix a bug to get multiple sequential tool calls to work * Cleanup * Cleanup and formatting * Cleanup and refactoring * Refactoring * More refactoring * More refactoring and updated _get_tool_calls_and_descriptions properly creates the final set of tool arguments * Refactoring based on comments * More cleanup * Update example * Formatting * fix license header * Fix sede bug * PR comments and simplification * Do more monkey patching * PR comments * PR comments * Refactoring and cleanup of utilities in strategies.py * Fix issue. Use tool name as fall back to match tool execution decision to tool call * ignore pylint import issues * Resolve todo * Two tool call question now works as expected * Fixing typing and formatting * Formatting * Update readme and docs * remove tool invoker docs since not needed anymore * Add some integration tests for Agent using confirmation strategies * Add more unit tests * Update haystack_experimental/components/agents/human_in_the_loop/breakpoint.py Co-authored-by: Stefano Fiorucci <[email protected]> * Refactoring based on PR comments * formatting and types --------- Co-authored-by: Stefano Fiorucci <[email protected]> * Add tests for strategies.py * More tests and slight refactoring * fix test * Add confirmation strategies to run_async in Agent and add integration tests * More tests for user interfaces * Add more tests * More tests --------- Co-authored-by: Stefano Fiorucci <[email protected]>
1 parent c554cc5 commit 5bc99c2

29 files changed

+3624
-13
lines changed

‎README.md‎

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -41,17 +41,18 @@ that includes it. Once it reaches the end of its lifespan, the experiment will b
4141

4242
### Active experiments
4343

44-
| Name | Type | Expected End Date | Dependencies | Cookbook | Discussion |
45-
|---------------------------------------|--------------------------------|-------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
46-
| [`InMemoryChatMessageStore`][1] | Memory Store | December 2024 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4] |
47-
| [`ChatMessageRetriever`][2] | Memory Component | December 2024 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4] |
48-
| [`ChatMessageWriter`][3] | Memory Component | December 2024 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4] |
49-
| [`QueryExpander`][5] | Query Expansion Component | October 2025 | None | None | [Discuss][6] |
50-
| [`EmbeddingBasedDocumentSplitter`][8] | EmbeddingBasedDocumentSplitter | August 2025 | None | None | [Discuss][7] |
51-
| [`MultiQueryEmbeddingRetriever`][13] | MultiQueryEmbeddingRetriever | November 2025 | None | None | [Discuss][11] |
52-
| [`MultiQueryTextRetriever`][14] | MultiQueryTextRetriever | November 2025 | None | None | [Discuss][12] |
53-
| [`OpenAIChatGenerator`][9] | Chat Generator Component | November 2025 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/hallucination_score_calculator.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][10] |
54-
| [`MarkdownHeaderLevelInferrer`][15] | Preprocessor | January 2025 | None | None | [Discuss][16] |
44+
| Name | Type | Expected End Date | Dependencies | Cookbook | Discussion |
45+
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------|-------------------|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
46+
| [`InMemoryChatMessageStore`][1] | Memory Store | December 2024 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4] |
47+
| [`ChatMessageRetriever`][2] | Memory Component | December 2024 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4] |
48+
| [`ChatMessageWriter`][3] | Memory Component | December 2024 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4] |
49+
| [`QueryExpander`][5] | Query Expansion Component | October 2025 | None | None | [Discuss][6] |
50+
| [`EmbeddingBasedDocumentSplitter`][8] | EmbeddingBasedDocumentSplitter | August 2025 | None | None | [Discuss][7] |
51+
| [`MultiQueryEmbeddingRetriever`][13] | MultiQueryEmbeddingRetriever | November 2025 | None | None | [Discuss][11] |
52+
| [`MultiQueryTextRetriever`][14] | MultiQueryTextRetriever | November 2025 | None | None | [Discuss][12] |
53+
| [`OpenAIChatGenerator`][9] | Chat Generator Component | November 2025 | None | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/hallucination_score_calculator.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][10] |
54+
| [`MarkdownHeaderLevelInferrer`][15] | Preprocessor | January 2025 | None | None | [Discuss][16] |
55+
| [`Agent`][17]; [Confirmation Policies][18]; [ConfirmationUIs][19]; [ConfirmationStrategies][20]; [`ConfirmationUIResult` and `ToolExecutionDecision`][21] [HITLBreakpointException][22] | Human in the Loop | December 2025 | rich | None | [Discuss][23] |
5556

5657
[1]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/chat_message_stores/in_memory.py
5758
[2]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/chat_message_retriever.py
@@ -69,8 +70,13 @@ that includes it. Once it reaches the end of its lifespan, the experiment will b
6970
[14]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/multi_query_text_retriever.py
7071
[15]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/preprocessors/md_header_level_inferrer.py
7172
[16]: https://github.com/deepset-ai/haystack-experimental/discussions/376
72-
73-
73+
[17]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/agents/agent.py
74+
[18]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/agents/human_in_the_loop/policies.py
75+
[19]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/agents/human_in_the_loop/user_interfaces.py
76+
[20]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/agents/human_in_the_loop/strategies.py
77+
[21]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/agents/human_in_the_loop/dataclasses.py
78+
[22]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/agents/human_in_the_loop/errors.py
79+
[23]: https://github.com/deepset-ai/haystack-experimental/discussions/XXX
7480

7581
### Adopted experiments
7682
| Name | Type | Final release |

‎docs/pydoc/config/agents_api.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
loaders:
2+
- type: haystack_pydoc_tools.loaders.CustomPythonLoader
3+
search_path: [../../../]
4+
modules: [
5+
"haystack_experimental.components.agents.agent",
6+
"haystack_experimental.components.agents.human_in_the_loop.breakpoint",
7+
"haystack_experimental.components.agents.human_in_the_loop.dataclasses",
8+
"haystack_experimental.components.agents.human_in_the_loop.errors",
9+
"haystack_experimental.components.agents.human_in_the_loop.policies",
10+
"haystack_experimental.components.agents.human_in_the_loop.strategies",
11+
"haystack_experimental.components.agents.human_in_the_loop.types",
12+
"haystack_experimental.components.agents.human_in_the_loop.user_interfaces",
13+
]
14+
ignore_when_discovered: ["__init__"]
15+
processors:
16+
- type: filter
17+
expression:
18+
documented_only: true
19+
do_not_filter_modules: false
20+
skip_empty_modules: true
21+
- type: smart
22+
- type: crossref
23+
renderer:
24+
type: haystack_pydoc_tools.renderers.ReadmeCoreRenderer
25+
excerpt: Tool-using agents with provider-agnostic chat model support.
26+
category_slug: haystack-api
27+
title: Agents
28+
slug: experimental-agents-api
29+
order: 2
30+
markdown:
31+
descriptive_class_title: false
32+
classdef_code_block: false
33+
descriptive_module_title: true
34+
add_method_class_prefix: true
35+
add_member_class_prefix: false
36+
filename: experimental_agents_api.md

‎haystack_experimental/components/agents/__init__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# SPDX-FileCopyrightText: 2022-present deepset GmbH <[email protected]>
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
import sys
6+
from typing import TYPE_CHECKING
7+
8+
from lazy_imports import LazyImporter
9+
10+
_import_structure = {"agent": ["Agent"]}
11+
12+
if TYPE_CHECKING:
13+
from .agent import Agent as Agent
14+
15+
else:
16+
sys.modules[__name__] = LazyImporter(name=__name__, module_file=__file__, import_structure=_import_structure)

0 commit comments

Comments
 (0)