Skip to content

Commit ed3f0d2

Browse files
garybadwaldavidsbatistampangrazzi
authored
feat: add CometAPI integration with CometAPIChatGenerator class and configuration (#2345)
* feat: Add GoogleAITextEmbedder and GoogleAIDocumentEmbedder components * fix: Improve error messages for input type validation in GoogleAITextEmbedder and GoogleAIDocumentEmbedder * feat: add Google GenAI embedder components for document and text embeddings * feat: add unit tests for GoogleAIDocumentEmbedder and GoogleAITextEmbedder * refactor: clean up imports and improve list handling in GoogleAIDocumentEmbedder and GoogleAITextEmbedder tests * refactor: Rename classes and update imports for Google GenAI components * feat: Add additional modules for Google GenAI embedders in config * chore: add 'more-itertools' to lint environment dependencies * refactor: update GoogleGenAIDocumentEmbedder and GoogleGenAITextEmbedder to use private attributes for initialization * refactor: update _prepare_texts_to_embed to return a list instead of a dictionary * refactor: format code for better readability and consistency in document embedder * refactor: improve code formatting for consistency and readability in document embedder and tests * refactor: update _prepare_texts_to_embed to return a list instead of a dictionary * feat: add new author to project metadata in pyproject.toml * feat: add asynchronous embedding methods for GoogleGenAIDocumentEmbedder and GoogleGenAITextEmbedder * fix: ensure consistent formatting for pylint * fix: update return type annotation for run_async method in GoogleGenAIDocumentEmbedder * fix: update return type annotation for run_async method in GoogleGenAITextEmbedder * fix: update return type annotation and handle None values in _embed_batch_async method * fix: remove unnecessary blank line in _embed_batch_async method * feat: add CometAPI integration with CometAPIChatGenerator class and configuration * feat: add CometAPI integration with chat generator and tests * feat: add CometAPI integration with workflow and labeler configuration * feat: add CometAPI integration to the inventory in README * feat: refactor CometAPIChatGenerator initialization and update test for async run * feat: add pytz dependency to test environment * feat: clean up CometAPIChatGenerator constructor formatting and add blank line in test file * feat: update type hints for streaming_callback and tools in CometAPIChatGenerator * feat: update streaming_callback type hint to use StreamingCallbackT for CometAPIChatGenerator * feat: remove unused Awaitable and Callable imports in chat_generator.py * feat: add initial Changelog file for CometAPI integration * feat: add pytz dependency to test environment and update mypy configuration * feat: remove unused mock_async_chat_completion fixture from test_run_async * feat: update mypy configuration to ignore missing imports for jsonref and add new linting rules * feat: add type hints for api_key and model parameters in CometAPIChatGenerator * style: format parameter definitions in CometAPIChatGenerator constructor * docs: add CometAPI resources section to README * docs: enhance usage description in README for CometAPI integration * Refactor CometAPIChatGenerator tests: consolidate async tests into a single file, update to synchronous methods, and enhance test coverage for initialization, running with parameters, and tool integration. * refactor: simplify type declarations in TestCometAPIChatGenerator tests * fix: correct function reference in TestCometAPIChatGenerator * refactor: remove redundant YAML serialization/deserialization in TestCometAPIChatGenerator * Fixed the type issue * Fixes for: 1. enforcing keyword arguments for all init params 2. updated workflow * Fixed the test-cases * Fixed the linting issue * Fix test-cases * Fix test cases * Fixed formating * Fixed test case * Updated Readme.md * Updated README.md * remove redundant pytestmark assignment in test_cometapi_chat_generator_async.py --------- Co-authored-by: David S. Batista <[email protected]> Co-authored-by: Michele Pangrazzi <[email protected]>
1 parent 76f21a6 commit ed3f0d2

File tree

13 files changed

+1708
-0
lines changed

13 files changed

+1708
-0
lines changed

.github/labeler.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@ integration:cohere:
3939
- any-glob-to-any-file: "integrations/cohere/**/*"
4040
- any-glob-to-any-file: ".github/workflows/cohere.yml"
4141

42+
integration:cometapi:
43+
- changed-files:
44+
- any-glob-to-any-file: "integrations/cometapi/**/*"
45+
- any-glob-to-any-file: ".github/workflows/cometapi.yml"
46+
4247
integration:deepeval:
4348
- changed-files:
4449
- any-glob-to-any-file: "integrations/deepeval/**/*"

.github/workflows/cometapi.yml

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# This workflow comes from https://github.com/ofek/hatch-mypyc
2+
# https://github.com/ofek/hatch-mypyc/blob/5a198c0ba8660494d02716cfc9d79ce4adfb1442/.github/workflows/test.yml
3+
name: Test / cometapi
4+
5+
on:
6+
schedule:
7+
- cron: "0 0 * * *"
8+
pull_request:
9+
paths:
10+
- "integrations/cometapi/**"
11+
- "!integrations/cometapi/*.md"
12+
- ".github/workflows/cometapi.yml"
13+
14+
defaults:
15+
run:
16+
working-directory: integrations/cometapi
17+
18+
concurrency:
19+
group: cometapi-${{ github.head_ref }}
20+
cancel-in-progress: true
21+
22+
env:
23+
PYTHONUNBUFFERED: "1"
24+
FORCE_COLOR: "1"
25+
COMET_API_KEY: "${{ secrets.COMET_API_KEY }}"
26+
27+
jobs:
28+
run:
29+
name: Python ${{ matrix.python-version }} on ${{ startsWith(matrix.os, 'macos-') && 'macOS' || startsWith(matrix.os, 'windows-') && 'Windows' || 'Linux' }}
30+
runs-on: ${{ matrix.os }}
31+
strategy:
32+
fail-fast: false
33+
matrix:
34+
os: [ubuntu-latest, windows-latest, macos-latest]
35+
python-version: ["3.9", "3.13"]
36+
max-parallel: 2 # to avoid "429 Resource has been exhausted"
37+
38+
steps:
39+
- name: Support longpaths
40+
if: matrix.os == 'windows-latest'
41+
working-directory: .
42+
run: git config --system core.longpaths true
43+
44+
- uses: actions/checkout@v5
45+
46+
- name: Set up Python ${{ matrix.python-version }}
47+
uses: actions/setup-python@v6
48+
with:
49+
python-version: ${{ matrix.python-version }}
50+
51+
- name: Install Hatch
52+
run: pip install --upgrade hatch
53+
- name: Lint
54+
if: matrix.python-version == '3.9' && runner.os == 'Linux'
55+
run: hatch run fmt-check && hatch run test:types
56+
57+
- name: Generate docs
58+
if: matrix.python-version == '3.9' && runner.os == 'Linux'
59+
run: hatch run docs
60+
61+
- name: Run tests
62+
run: hatch run test:cov-retry
63+
64+
- name: Run unit tests with lowest direct dependencies
65+
run: |
66+
hatch run uv pip compile pyproject.toml --resolution lowest-direct --output-file requirements_lowest_direct.txt
67+
hatch -e test env run -- uv pip install -r requirements_lowest_direct.txt
68+
hatch run test:unit
69+
70+
- name: Nightly - run unit tests with Haystack main branch
71+
if: github.event_name == 'schedule'
72+
run: |
73+
hatch env prune
74+
hatch run uv pip install git+https://github.com/deepset-ai/haystack.git@main
75+
hatch run test:unit
76+
77+
- name: Send event to Datadog for nightly failures
78+
if: failure() && github.event_name == 'schedule'
79+
uses: ./.github/actions/send_failure
80+
with:
81+
title: |
82+
Core integrations nightly tests failure: ${{ github.workflow }}
83+
api-key: ${{ secrets.CORE_DATADOG_API_KEY }}

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ Please check out our [Contribution Guidelines](CONTRIBUTING.md) for all the deta
3232
| [azure-ai-search-haystack](integrations/azure_ai_search/) | Document Store | [![PyPI - Version](https://img.shields.io/pypi/v/azure-ai-search-haystack.svg)](https://pypi.org/project/azure-ai-search-haystack) | [![Test / azure-ai-search](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/azure_ai_search.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/azure_ai_search.yml) |
3333
| [chroma-haystack](integrations/chroma/) | Document Store | [![PyPI - Version](https://img.shields.io/pypi/v/chroma-haystack.svg)](https://pypi.org/project/chroma-haystack) | [![Test / chroma](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/chroma.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/chroma.yml) |
3434
| [cohere-haystack](integrations/cohere/) | Embedder, Generator, Ranker | [![PyPI - Version](https://img.shields.io/pypi/v/cohere-haystack.svg)](https://pypi.org/project/cohere-haystack) | [![Test / cohere](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/cohere.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/cohere.yml) |
35+
| [cometapi-haystack](integrations/cometapi/) | Embedder, Generator, Ranker | [![PyPI - Version](https://img.shields.io/pypi/v/cometapi-haystack.svg)](https://pypi.org/project/cometapi-haystack) | [![Test / cometapi](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/cometapi.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/cometapi.yml) |
3536
| [deepeval-haystack](integrations/deepeval/) | Evaluator | [![PyPI - Version](https://img.shields.io/pypi/v/deepeval-haystack.svg)](https://pypi.org/project/deepeval-haystack) | [![Test / deepeval](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/deepeval.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/deepeval.yml) |
3637
| [elasticsearch-haystack](integrations/elasticsearch/) | Document Store | [![PyPI - Version](https://img.shields.io/pypi/v/elasticsearch-haystack.svg)](https://pypi.org/project/elasticsearch-haystack) | [![Test / elasticsearch](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/elasticsearch.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/elasticsearch.yml) |
3738
| [fastembed-haystack](integrations/fastembed/) | Embedder, Ranker | [![PyPI - Version](https://img.shields.io/pypi/v/fastembed-haystack.svg)](https://pypi.org/project/fastembed-haystack/) | [![Test / fastembed](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/fastembed.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/fastembed.yml) |

integrations/cometapi/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Changelog

integrations/cometapi/LICENSE.txt

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
Apache License
2+
Version 2.0, January 2004
3+
http://www.apache.org/licenses/
4+
5+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6+
7+
1. Definitions.
8+
9+
"License" shall mean the terms and conditions for use, reproduction,
10+
and distribution as defined by Sections 1 through 9 of this document.
11+
12+
"Licensor" shall mean the copyright owner or entity granting the License.
13+
14+
"Legal Entity" shall mean the union of the acting entity and all
15+
other entities that control, are controlled by, or are under common
16+
control with that entity. For the purposes of this definition,
17+
"control" means (i) the power, direct or indirect, to cause the
18+
direction or management of such entity, whether by contract or
19+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
20+
outstanding shares, or (iii) beneficial ownership of such entity.
21+
22+
"You" (or "Your") shall mean an individual or Legal Entity
23+
exercising permissions granted by this License.
24+
25+
"Source" form shall mean the preferred form for making modifications,
26+
including but not limited to software source code, documentation
27+
source, and configuration files.
28+
29+
"Object" form shall mean any form resulting from mechanical
30+
transformation or translation of a Source form, including but
31+
not limited to compiled object code, generated documentation,
32+
and conversions to other media types.
33+
34+
"Work" shall mean the work of authorship, whether in Source or
35+
Object form, made available under the License, as indicated by a
36+
copyright notice that is included in or attached to the work
37+
(which shall not include communications that are marked or
38+
designated in writing by the copyright owner as "Not a Contribution").
39+
40+
"Contribution" shall mean any work of authorship, including
41+
the original version of the Work and any modifications or additions
42+
to that Work or Derivative Works thereof, that is intentionally
43+
submitted to Licensor for inclusion in the Work by the copyright owner
44+
or by an individual or Legal Entity authorized to submit on behalf of
45+
the copyright owner. For the purposes of this definition, "submitted"
46+
means any form of electronic, verbal, or written communication sent
47+
to the Licensor or its representatives, including but not limited to
48+
communication on electronic mailing lists, source code control
49+
systems, and issue tracking systems that are managed by, or on behalf
50+
of, the Licensor for the purpose of discussing and improving the Work,
51+
but excluding communication that is conspicuously marked or otherwise
52+
designated in writing by the copyright owner as "Not a Contribution".
53+
54+
"Contributor" shall mean Licensor and any individual or Legal Entity
55+
on behalf of whom a Contribution has been received by Licensor and
56+
subsequently incorporated within the Work.
57+
58+
2. Grant of Copyright License. Subject to the terms and conditions of
59+
this License, each Contributor hereby grants to You a perpetual,
60+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
61+
copyright license to use, reproduce, modify, distribute, and prepare
62+
Derivative Works of, publicly display, publicly perform, sublicense,
63+
and distribute the Work and such Derivative Works in Source or Object
64+
form.
65+
66+
3. Grant of Patent License. Subject to the terms and conditions of
67+
this License, each Contributor hereby grants to You a perpetual,
68+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69+
(except as stated in this section) patent license to make, have made,
70+
use, offer to sell, sell, import, and otherwise transfer the Work,
71+
where such license applies only to those patent claims licensable
72+
by such Contributor that are necessarily infringed by their
73+
Contribution(s) alone or by combination of their Contribution(s)
74+
with the Work to which such Contribution(s) was submitted. If You
75+
institute patent litigation against any entity (including a
76+
cross-claim or counterclaim in a lawsuit) alleging that the Work
77+
or a Contribution incorporated within the Work constitutes direct
78+
or contributory patent infringement, then any patent licenses
79+
granted to You under this License for that Work shall terminate
80+
as of the date such litigation is filed.
81+
82+
4. Redistribution. You may reproduce and distribute copies of the
83+
Work or Derivative Works thereof in any medium, with or without
84+
modifications, and in Source or Object form, provided that You
85+
meet the following conditions:
86+
87+
(a) You must give any other recipients of the Work or
88+
Derivative Works a copy of this License; and
89+
90+
(b) You must cause any modified files to carry prominent notices
91+
stating that You changed the files; and
92+
93+
(c) You must retain, in the Source form of any Derivative Works
94+
that You distribute, all copyright, trademark, patent,
95+
attribution and other notices from the Source form of the Work,
96+
excluding those notices that do not pertain to any part of
97+
the Derivative Works; and
98+
99+
(d) If the Work includes a "NOTICE" text file as part of its
100+
distribution, then any Derivative Works that You distribute must
101+
include a readable copy of the attribution notices contained
102+
within such NOTICE file, excluding those notices that do not
103+
pertain to any part of the Derivative Works, in at least one
104+
of the following places: within a NOTICE text file distributed
105+
as part of the Derivative Works; within the Source form or
106+
documentation, if provided along with the Derivative Works; or,
107+
within a display generated by the Derivative Works, if and
108+
wherever such third-party notices normally appear. The contents
109+
of the NOTICE file are for informational purposes only and
110+
do not modify the License. You may add Your own attribution
111+
notices within Derivative Works that You distribute, alongside
112+
or as an addendum to the NOTICE text from the Work, provided
113+
that such additional attribution notices cannot be construed
114+
as modifying the License.
115+
116+
You may add Your own copyright notice to Your modifications and
117+
may provide additional or different license terms and conditions
118+
for use, reproduction, or distribution of Your modifications, or
119+
for any such Derivative Works as a whole, provided Your use,
120+
reproduction, and distribution of the Work otherwise complies with
121+
the conditions stated in this License.
122+
123+
5. Submission of Contributions. Unless You explicitly state otherwise,
124+
any Contribution intentionally submitted for inclusion in the Work
125+
by You to the Licensor shall be under the terms and conditions of
126+
this License, without any additional terms or conditions.
127+
Notwithstanding the above, nothing herein shall supersede or modify
128+
the terms of any separate license agreement you may have executed
129+
with Licensor regarding such Contributions.
130+
131+
6. Trademarks. This License does not grant permission to use the trade
132+
names, trademarks, service marks, or product names of the Licensor,
133+
except as required for reasonable and customary use in describing the
134+
origin of the Work and reproducing the content of the NOTICE file.
135+
136+
7. Disclaimer of Warranty. Unless required by applicable law or
137+
agreed to in writing, Licensor provides the Work (and each
138+
Contributor provides its Contributions) on an "AS IS" BASIS,
139+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
140+
implied, including, without limitation, any warranties or conditions
141+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
142+
PARTICULAR PURPOSE. You are solely responsible for determining the
143+
appropriateness of using or redistributing the Work and assume any
144+
risks associated with Your exercise of permissions under this License.
145+
146+
8. Limitation of Liability. In no event and under no legal theory,
147+
whether in tort (including negligence), contract, or otherwise,
148+
unless required by applicable law (such as deliberate and grossly
149+
negligent acts) or agreed to in writing, shall any Contributor be
150+
liable to You for damages, including any direct, indirect, special,
151+
incidental, or consequential damages of any character arising as a
152+
result of this License or out of the use or inability to use the
153+
Work (including but not limited to damages for loss of goodwill,
154+
work stoppage, computer failure or malfunction, or any and all
155+
other commercial damages or losses), even if such Contributor
156+
has been advised of the possibility of such damages.
157+
158+
9. Accepting Warranty or Support. You may choose to offer, and to
159+
charge a fee for, warranty, support, indemnity or other liability
160+
obligations and/or rights consistent with this License. However, in
161+
accepting such obligations, You may act only on Your own behalf and on
162+
Your sole responsibility, not on behalf of any other Contributor, and
163+
only if You agree to indemnify, defend, and hold each Contributor
164+
harmless for any liability incurred by, or claims asserted against,
165+
such Contributor by reason of your accepting any such warranty or support.
166+
167+
END OF TERMS AND CONDITIONS
168+
169+
APPENDIX: How to apply the Apache License to your work.
170+
171+
To apply the Apache License to your work, attach the following
172+
boilerplate notice, with the fields enclosed by brackets "[]"
173+
replaced with your own identifying information. (Don't include
174+
the brackets!) The text should be enclosed in the appropriate
175+
comment syntax for the file format. We also recommend that a
176+
file or class name and description of purpose be included on the
177+
same page as the copyright notice for easier identification within
178+
third-party archives.
179+
180+
Copyright 2023-present deepset GmbH
181+
182+
Licensed under the Apache License, Version 2.0 (the "License");
183+
you may not use this file except in compliance with the License.
184+
You may obtain a copy of the License at
185+
186+
http://www.apache.org/licenses/LICENSE-2.0
187+
188+
Unless required by applicable law or agreed to in writing, software
189+
distributed under the License is distributed on an "AS IS" BASIS,
190+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
191+
See the License for the specific language governing permissions and
192+
limitations under the License.

integrations/cometapi/README.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# cometapi-haystack
2+
3+
[![PyPI - Version](https://img.shields.io/pypi/v/cometapi-haystack.svg)](https://pypi.org/project/cometapi-haystack)
4+
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/cometapi-haystack.svg)](https://pypi.org/project/cometapihaystack)
5+
6+
- [Integration page](https://haystack.deepset.ai/integrations/comet-api)
7+
- [Changelog](https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/cometapi/CHANGELOG.md)
8+
9+
---
10+
11+
## Contributing
12+
13+
Refer to the general [Contribution Guidelines](https://github.com/deepset-ai/haystack-core-integrations/blob/main/CONTRIBUTING.md).
14+
15+
To run integration tests locally, you need to export the `COMET_API_KEY` environment variable.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
loaders:
2+
- type: haystack_pydoc_tools.loaders.CustomPythonLoader
3+
search_path: [../src]
4+
modules: [
5+
"haystack_integrations.components.generators.cometapi.chat.chat_generator",
6+
]
7+
ignore_when_discovered: ["__init__"]
8+
processors:
9+
- type: filter
10+
expression:
11+
documented_only: true
12+
do_not_filter_modules: false
13+
skip_empty_modules: true
14+
- type: smart
15+
- type: crossref
16+
renderer:
17+
type: haystack_pydoc_tools.renderers.ReadmeIntegrationRenderer
18+
excerpt: Comet API integration for Haystack
19+
category_slug: integrations-api
20+
title: Comet API
21+
slug: integrations-cometapi
22+
order: 91
23+
markdown:
24+
descriptive_class_title: false
25+
classdef_code_block: false
26+
descriptive_module_title: true
27+
add_method_class_prefix: true
28+
add_member_class_prefix: false
29+
filename: _readme_cometapi.md

0 commit comments

Comments
 (0)