diff --git a/.changeset/clear-suns-sleep.md b/.changeset/clear-suns-sleep.md new file mode 100644 index 000000000..ad2cee685 --- /dev/null +++ b/.changeset/clear-suns-sleep.md @@ -0,0 +1,5 @@ +--- +'@sap-ai-sdk/langchain': minor +--- + +[New Functionality] Add LangChain Orchestration client. diff --git a/packages/langchain/README.md b/packages/langchain/README.md index 869a99ce0..376f2e776 100644 --- a/packages/langchain/README.md +++ b/packages/langchain/README.md @@ -2,17 +2,15 @@ SAP Cloud SDK for AI is the official Software Development Kit (SDK) for **SAP AI Core**, **SAP Generative AI Hub**, and **Orchestration Service**. -This package provides LangChain model clients built on top of the foundation model clients of the SAP Cloud SDK for AI. +This package provides LangChain clients built on top of the foundation model and orchestration clients of the SAP Cloud SDK for AI. ### Table of Contents - [Installation](#installation) - [Prerequisites](#prerequisites) -- [Relationship between Models and Deployment ID](#relationship-between-models-and-deployment-id) - [Usage](#usage) - - [Client Initialization](#client-initialization) - - [Chat Client](#chat-client) - - [Embedding Client](#embedding-client) + - [Orchestration Client](#orchestration-client) + - [Azure OpenAI Client](#azure-openai-client) - [Local Testing](#local-testing) - [Support, Feedback, Contribution](#support-feedback-contribution) - [License](#license) @@ -28,10 +26,11 @@ $ npm install @sap-ai-sdk/langchain - [Enable the AI Core service in SAP BTP](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/initial-setup). - Use the same `@langchain/core` version as the `@sap-ai-sdk/langchain` package, to see which langchain version this package is currently using, check our [package.json](./package.json). - Configure the project with **Node.js v20 or higher** and **native ESM** support. -- Ensure a deployed OpenAI model is available in the SAP Generative AI Hub. - - Use the [`DeploymentApi`](https://github.com/SAP/ai-sdk-js/blob/main/packages/ai-api/README.md#create-a-deployment) from `@sap-ai-sdk/ai-api` [to deploy a model](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-generative-ai-model-in-sap-ai-core). - Alternatively, you can also create deployments using the [SAP AI Launchpad](https://help.sap.com/docs/sap-ai-core/generative-ai-hub/activate-generative-ai-hub-for-sap-ai-launchpad?locale=en-US&q=launchpad). - - Once deployment is complete, access the model via the `deploymentUrl`. +- Ensure that a relevant deployment is available in the SAP Generative AI Hub: + - Use the [`DeploymentApi`](https://github.com/SAP/ai-sdk-js/blob/main/packages/ai-api/README.md#create-a-deployment) from `@sap-ai-sdk/ai-api` or the [SAP AI Launchpad](https://help.sap.com/docs/sap-ai-core/generative-ai-hub/activate-generative-ai-hub-for-sap-ai-launchpad?locale=en-US&q=launchpad) to create a deployment. + - For **OpenAI model**, follow [this guide](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-generative-ai-model-in-sap-ai-core). + - For **orchestration service**, follow [this guide](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-orchestration). + - Once deployed, access the service via the `deploymentUrl`. > **Accessing the AI Core Service via the SDK** > @@ -40,168 +39,18 @@ $ npm install @sap-ai-sdk/langchain > - In Cloud Foundry, it's accessed from the `VCAP_SERVICES` environment variable. > - In Kubernetes / Kyma environments, you have to mount the service binding as a secret instead, for more information refer to [this documentation](https://www.npmjs.com/package/@sap/xsenv#usage-in-kubernetes). -## Relationship between Models and Deployment ID - -SAP AI Core manages access to generative AI models through the global AI scenario `foundation-models`. -Creating a deployment for a model requires access to this scenario. - -Each model, model version, and resource group allows for a one-time deployment. -After deployment completion, the response includes a `deploymentUrl` and an `id`, which is the deployment ID. -For more information, see [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-generative-ai-model-in-sap-ai-core). - -[Resource groups](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/resource-groups?q=resource+group) represent a virtual collection of related resources within the scope of one SAP AI Core tenant. - -Consequently, each deployment ID and resource group uniquely map to a combination of model and model version within the `foundation-models` scenario. - ## Usage -This package offers both chat and embedding clients, currently supporting Azure OpenAI. +This package offers LangChain clients for Azure OpenAI and SAP Orchestration service. All clients comply with [LangChain's interface](https://js.langchain.com/docs/introduction). -### Client Initialization - -To initialize a client, provide the model name: - -```ts -import { - AzureOpenAiChatClient, - AzureOpenAiEmbeddingClient -} from '@sap-ai-sdk/langchain'; - -// For a chat client -const chatClient = new AzureOpenAiChatClient({ modelName: 'gpt-4o' }); -// For an embedding client -const embeddingClient = new AzureOpenAiEmbeddingClient({ modelName: 'gpt-4o' }); -``` - -In addition to the default parameters of the model vendor (e.g., OpenAI) and LangChain, additional parameters can be used to help narrow down the search for the desired model: - -```ts -const chatClient = new AzureOpenAiChatClient({ - modelName: 'gpt-4o', - modelVersion: '24-07-2021', - resourceGroup: 'my-resource-group' -}); -``` - -**Do not pass a `deployment ID` to initialize the client.** -For the LangChain model clients, initialization is done using the model name, model version and resource group. - -An important note is that LangChain clients by default attempt 6 retries with exponential backoff in case of a failure. -Especially in testing environments you might want to reduce this number to speed up the process: - -```ts -const embeddingClient = new AzureOpenAiEmbeddingClient({ - modelName: 'gpt-4o', - maxRetries: 0 -}); -``` - -#### Custom Destination - -When initializing the `AzureOpenAiChatClient` and `AzureOpenAiEmbeddingClient` clients, it is possible to provide a custom destination. -For example, when targeting a destination with the name `my-destination`, the following code can be used: - -```ts -const chatClient = new AzureOpenAiChatClient( - { - modelName: 'gpt-4o', - modelVersion: '24-07-2021', - resourceGroup: 'my-resource-group' - }, - { - destinationName: 'my-destination' - } -); -``` - -By default, the fetched destination is cached. -To disable caching, set the `useCache` parameter to `false` together with the `destinationName` parameter. +### Orchestration Client -### Chat Client +For more information about the Orchestration client, refer to the [documentation](https://github.com/SAP/ai-sdk-js/tree/main/packages/langchain/src/orchestration/README.md). -The chat client allows you to interact with Azure OpenAI chat models, accessible via the generative AI hub of SAP AI Core. -To invoke the client, pass a prompt: +### Azure OpenAI Client -```ts -const response = await chatClient.invoke("What's the capital of France?"); -``` - -#### Advanced Example with Templating and Output Parsing - -```ts -import { AzureOpenAiChatClient } from '@sap-ai-sdk/langchain'; -import { StringOutputParser } from '@langchain/core/output_parsers'; -import { ChatPromptTemplate } from '@langchain/core/prompts'; - -// initialize the client -const client = new AzureOpenAiChatClient({ modelName: 'gpt-35-turbo' }); - -// create a prompt template -const promptTemplate = ChatPromptTemplate.fromMessages([ - ['system', 'Answer the following in {language}:'], - ['user', '{text}'] -]); -// create an output parser -const parser = new StringOutputParser(); - -// chain together template, client, and parser -const llmChain = promptTemplate.pipe(client).pipe(parser); - -// invoke the chain -return llmChain.invoke({ - language: 'german', - text: 'What is the capital of France?' -}); -``` - -### Embedding Client - -Embedding clients allow embedding either text or document chunks (represented as arrays of strings). -While you can use them standalone, they are usually used in combination with other LangChain utilities, like a text splitter for preprocessing and a vector store for storage and retrieval of the relevant embeddings. -For a complete example how to implement RAG with our LangChain client, take a look at our [sample code](https://github.com/SAP/ai-sdk-js/blob/main/sample-code/src/langchain-azure-openai.ts). - -#### Embed Text - -```ts -const embeddedText = await embeddingClient.embedQuery( - 'Paris is the capital of France.' -); -``` - -#### Embed Document Chunks - -```ts -const embeddedDocuments = await embeddingClient.embedDocuments([ - 'Page 1: Paris is the capital of France.', - 'Page 2: It is a beautiful city.' -]); -``` - -#### Preprocess, embed, and store documents - -```ts -// Create a text splitter and split the document -const textSplitter = new RecursiveCharacterTextSplitter({ - chunkSize: 2000, - chunkOverlap: 200 -}); -const splits = await textSplitter.splitDocuments(docs); - -// Initialize the embedding client -const embeddingClient = new AzureOpenAiEmbeddingClient({ - modelName: 'text-embedding-ada-002' -}); - -// Create a vector store from the document -const vectorStore = await MemoryVectorStore.fromDocuments( - splits, - embeddingClient -); - -// Create a retriever for the vector store -const retriever = vectorStore.asRetriever(); -``` +For more information about Azure OpenAI client, refer to the [documentation](https://github.com/SAP/ai-sdk-js/tree/main/packages/langchain/src/openai/README.md). ## Local Testing diff --git a/packages/langchain/package.json b/packages/langchain/package.json index fe86e1742..1a08af10b 100644 --- a/packages/langchain/package.json +++ b/packages/langchain/package.json @@ -29,6 +29,7 @@ "@sap-ai-sdk/ai-api": "workspace:^", "@sap-ai-sdk/core": "workspace:^", "@sap-ai-sdk/foundation-models": "workspace:^", + "@sap-ai-sdk/orchestration": "workspace:^", "@sap-cloud-sdk/connectivity": "^3.26.1", "uuid": "^11.1.0", "@langchain/core": "0.3.40", diff --git a/packages/langchain/src/index.ts b/packages/langchain/src/index.ts index 5e1fcb6ce..a5aa75dec 100644 --- a/packages/langchain/src/index.ts +++ b/packages/langchain/src/index.ts @@ -7,3 +7,5 @@ export type { AzureOpenAiEmbeddingModelParams, AzureOpenAiChatCallOptions } from './openai/index.js'; +export { OrchestrationClient } from './orchestration/index.js'; +export type { OrchestrationCallOptions } from './orchestration/index.js'; diff --git a/packages/langchain/src/openai/README.md b/packages/langchain/src/openai/README.md new file mode 100644 index 000000000..a101d948b --- /dev/null +++ b/packages/langchain/src/openai/README.md @@ -0,0 +1,183 @@ +# @sap-ai-sdk/langchain + +SAP Cloud SDK for AI is the official Software Development Kit (SDK) for **SAP AI Core**, **SAP Generative AI Hub**, and **Orchestration Service**. + +This package provides LangChain model clients built on top of the foundation model clients of the SAP Cloud SDK for AI. + +> **Note**: For installation and prerequisites, refer to the [README](../../README.md). + +### Table of Contents + +- [Relationship between Models and Deployment ID](#relationship-between-models-and-deployment-id) +- [Usage](#usage) + - [Client Initialization](#client-initialization) + - [Chat Client](#chat-client) + - [Embedding Client](#embedding-client) +- [Local Testing](#local-testing) + +## Relationship between Models and Deployment ID + +SAP AI Core manages access to generative AI models through the global AI scenario `foundation-models`. +Creating a deployment for a model requires access to this scenario. + +Each model, model version, and resource group allows for a one-time deployment. +After deployment completion, the response includes a `deploymentUrl` and an `id`, which is the deployment ID. +For more information, see [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-generative-ai-model-in-sap-ai-core). + +[Resource groups](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/resource-groups?q=resource+group) represent a virtual collection of related resources within the scope of one SAP AI Core tenant. + +Consequently, each deployment ID and resource group uniquely map to a combination of model and model version within the `foundation-models` scenario. + +## Usage + +This package offers both chat and embedding clients for Azure OpenAI. +The client complies with [LangChain's interface](https://js.langchain.com/docs/introduction). + +### Client Initialization + +To initialize a client, provide the model name: + +```ts +import { + AzureOpenAiChatClient, + AzureOpenAiEmbeddingClient +} from '@sap-ai-sdk/langchain'; + +// For a chat client +const chatClient = new AzureOpenAiChatClient({ modelName: 'gpt-4o' }); +// For an embedding client +const embeddingClient = new AzureOpenAiEmbeddingClient({ modelName: 'gpt-4o' }); +``` + +In addition to the default parameters of Azure OpenAI and LangChain, additional parameters can be used to help narrow down the search for the desired model: + +```ts +const chatClient = new AzureOpenAiChatClient({ + modelName: 'gpt-4o', + modelVersion: '24-07-2021', + resourceGroup: 'my-resource-group' +}); +``` + +**Do not pass a `deployment ID` to initialize the client.** +For LangChain model clients, initialization requires specifying the model name, model version, and resource group. + +By default, LangChain clients retry up to 6 times with exponential backoff in case of failure. +In testing environments, reducing this number can speed up the process: + +```ts +const embeddingClient = new AzureOpenAiEmbeddingClient({ + modelName: 'gpt-4o', + maxRetries: 0 +}); +``` + +#### Custom Destination + +When initializing the `AzureOpenAiChatClient` and `AzureOpenAiEmbeddingClient`, a custom destination can be specified. +For example, to target `my-destination`, use the following code: + +```ts +const chatClient = new AzureOpenAiChatClient( + { + modelName: 'gpt-4o', + modelVersion: '24-07-2021', + resourceGroup: 'my-resource-group' + }, + { + destinationName: 'my-destination' + } +); +``` + +By default, the fetched destination is cached. +To disable caching, set the `useCache` parameter to `false` together with the `destinationName` parameter. + +### Chat Client + +The `AzureOpenAiChatClient` allows interaction with Azure OpenAI chat models, accessible through the Generative AI Hub of SAP AI Core. +To invoke the client, pass a prompt as shown below: + +```ts +const response = await chatClient.invoke("What's the capital of France?"); +``` + +#### Advanced Example with Templating and Output Parsing + +```ts +import { AzureOpenAiChatClient } from '@sap-ai-sdk/langchain'; +import { StringOutputParser } from '@langchain/core/output_parsers'; +import { ChatPromptTemplate } from '@langchain/core/prompts'; + +// initialize the client +const client = new AzureOpenAiChatClient({ modelName: 'gpt-35-turbo' }); + +// create a prompt template +const promptTemplate = ChatPromptTemplate.fromMessages([ + ['system', 'Answer the following in {language}:'], + ['user', '{text}'] +]); +// create an output parser +const parser = new StringOutputParser(); + +// chain together template, client, and parser +const llmChain = promptTemplate.pipe(client).pipe(parser); + +// invoke the chain +return llmChain.invoke({ + language: 'german', + text: 'What is the capital of France?' +}); +``` + +### Embedding Client + +The `AzureOpenAiEmbeddingClient` allows embedding of text or document chunks (represented as arrays of strings). +While it can be used standalone, it is typically combined with other LangChain utilities, such as a text splitter for preprocessing and a vector store for storing and retrieving relevant embeddings. +For a complete example of how to implement RAG with the LangChain client, refer to the [sample code](https://github.com/SAP/ai-sdk-js/blob/main/sample-code/src/langchain-azure-openai.ts). + +#### Embed Text + +```ts +const embeddedText = await embeddingClient.embedQuery( + 'Paris is the capital of France.' +); +``` + +#### Embed Document Chunks + +```ts +const embeddedDocuments = await embeddingClient.embedDocuments([ + 'Page 1: Paris is the capital of France.', + 'Page 2: It is a beautiful city.' +]); +``` + +#### Preprocess, embed, and store documents + +```ts +// Create a text splitter and split the document +const textSplitter = new RecursiveCharacterTextSplitter({ + chunkSize: 2000, + chunkOverlap: 200 +}); +const splits = await textSplitter.splitDocuments(docs); + +// Initialize the embedding client +const embeddingClient = new AzureOpenAiEmbeddingClient({ + modelName: 'text-embedding-ada-002' +}); + +// Create a vector store from the document +const vectorStore = await MemoryVectorStore.fromDocuments( + splits, + embeddingClient +); + +// Create a retriever for the vector store +const retriever = vectorStore.asRetriever(); +``` + +## Local Testing + +For local testing instructions, refer to this [section](https://github.com/SAP/ai-sdk-js/blob/main/README.md#local-testing). diff --git a/packages/langchain/src/orchestration/README.md b/packages/langchain/src/orchestration/README.md new file mode 100644 index 000000000..a23d11edf --- /dev/null +++ b/packages/langchain/src/orchestration/README.md @@ -0,0 +1,121 @@ +# @sap-ai-sdk/langchain + +SAP Cloud SDK for AI is the official Software Development Kit (SDK) for **SAP AI Core**, **SAP Generative AI Hub**, and **Orchestration Service**. + +This package provides LangChain model clients built on top of the foundation model clients of the SAP Cloud SDK for AI. + +> **Note**: For installation and prerequisites, refer to the [README](../../README.md). + +### Table of Contents + +- [Relationship between Orchestration and Resource Groups](#relationship-between-orchestration-and-resource-groups) +- [Usage](#usage) + - [Client Initialization](#client-initialization) + - [Client](#client) + - [Resilience](#resilience) +- [Local Testing](#local-testing) +- [Limitations](#limitations) + +## Relationship between Orchestration and Resource Groups + +SAP AI Core manages access to orchestration of generative AI models through the global AI scenario `orchestration`. +Creating a deployment for enabling orchestration capabilities requires access to this scenario. + +[Resource groups](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/resource-groups?q=resource+group) represent a virtual collection of related resources within the scope of one SAP AI Core tenant. +Each resource group allows for a one-time orchestration deployment. + +Consequently, each orchestration deployment uniquely maps to a resource group within the `orchestration` scenario. + +## Usage + +This package offers a LangChain orchestration service client. +Streaming is not supported. +The client complies with [LangChain's interface](https://js.langchain.com/docs/introduction). + +### Client Initialization + +To initialize the client, four different configurations can be provided. +The only required configuration is the orchestration configuration, explained in detail in the [orchestration foundation client](https://github.com/SAP/ai-sdk-js/blob/main/packages/orchestration/README.md). +Additionally, it is possible to set [default LangChain options](https://v03.api.js.langchain.com/types/_langchain_core.language_models_chat_models.BaseChatModelParams.html), a custom resource group, and a destination. + +A minimal example for instantiating the orchestration client uses a template and model name: + +```ts +import { OrchestrationClient } from '@sap-ai-sdk/langchain'; +const config: OrchestrationModuleConfig = { + llm: { + model_name: 'gpt-35-turbo' + }, + templating: { + template: [ + { role: 'user', content: 'Give me a long introduction of {{?subject}}' } + ] + } +}; + +const client = new OrchestrationClient(config); +``` + +#### Custom Destination + +The `OrchestrationClient` can be initialized with a custom destination. +For example, to target `my-destination`, use the following code: + +```ts +const client = new OrchestrationClient( + orchestrationConfig, + langchainOptions, + deploymentConfig, + { destinationName: 'my-destination' } +); +``` + +By default, the fetched destination is cached. +To disable caching, set the `useCache` parameter to `false` together with the `destinationName` parameter. + +### Client Invocation + +When invoking the client, pass a message history and, in most cases, input parameters for the template module. + +```ts +const systemMessage = new SystemMessage('Be a helpful assisstant!'); +const history = [systemMessage]; +const response = await client.invoke(history, { + inputParams: { subject: 'Paris' } +}); +``` + +#### Resilience + +To add resilience to the client, use LangChain's default options, especially `timeout` and `maxRetry`. + +##### Timeout + +By default, no timeout is set in the client. +To limit the maximum duration for the entire request, including retries, specify a timeout in milliseconds when using the `invoke` method: + +```ts +const response = await client.invoke(messageHistory, { timeout: 10000 }); +``` + +##### Retry + +LangChain clients retry up to 6 times by default. +To modify this behavior, set the `maxRetries` option during client initialization: + +```ts +const client = new OrchestrationClient(orchestrationConfig, { + maxRetries: 0 +}); +``` + +## Local Testing + +For local testing instructions, refer to this [section](https://github.com/SAP/ai-sdk-js/blob/main/README.md#local-testing). + +## Limitations + +Currently unsupported features are: + +- Streaming +- Tool Calling diff --git a/packages/langchain/src/orchestration/__snapshots__/client.test.ts.snap b/packages/langchain/src/orchestration/__snapshots__/client.test.ts.snap new file mode 100644 index 000000000..a2e3efc8d --- /dev/null +++ b/packages/langchain/src/orchestration/__snapshots__/client.test.ts.snap @@ -0,0 +1,79 @@ +// Jest Snapshot v1, https://goo.gl/fbAQLP + +exports[`orchestration service client returns successful response when maxRetries equals retry configuration 1`] = ` +{ + "id": [ + "langchain_core", + "messages", + "OrchestrationMessage", + ], + "kwargs": { + "additional_kwargs": { + "finish_reason": "stop", + "function_call": undefined, + "index": 0, + "tool_calls": undefined, + }, + "content": "Hello! How can I assist you today?", + "invalid_tool_calls": [], + "response_metadata": { + "created": 172, + "finish_reason": "stop", + "function_call": undefined, + "id": "orchestration-id", + "index": 0, + "model": "gpt-35-turbo", + "object": "chat.completion", + "system_fingerprint": undefined, + "tokenUsage": { + "completionTokens": 9, + "promptTokens": 9, + "totalTokens": 18, + }, + "tool_calls": undefined, + }, + "tool_calls": [], + }, + "lc": 1, + "type": "constructor", +} +`; + +exports[`orchestration service client returns successful response when timeout is bigger than delay 1`] = ` +{ + "id": [ + "langchain_core", + "messages", + "OrchestrationMessage", + ], + "kwargs": { + "additional_kwargs": { + "finish_reason": "stop", + "function_call": undefined, + "index": 0, + "tool_calls": undefined, + }, + "content": "Hello! How can I assist you today?", + "invalid_tool_calls": [], + "response_metadata": { + "created": 172, + "finish_reason": "stop", + "function_call": undefined, + "id": "orchestration-id", + "index": 0, + "model": "gpt-35-turbo", + "object": "chat.completion", + "system_fingerprint": undefined, + "tokenUsage": { + "completionTokens": 9, + "promptTokens": 9, + "totalTokens": 18, + }, + "tool_calls": undefined, + }, + "tool_calls": [], + }, + "lc": 1, + "type": "constructor", +} +`; diff --git a/packages/langchain/src/orchestration/client.test.ts b/packages/langchain/src/orchestration/client.test.ts new file mode 100644 index 000000000..791e19ead --- /dev/null +++ b/packages/langchain/src/orchestration/client.test.ts @@ -0,0 +1,106 @@ +import { constructCompletionPostRequest } from '@sap-ai-sdk/orchestration/internal.js'; +import { jest } from '@jest/globals'; +import nock from 'nock'; +import { + mockClientCredentialsGrantCall, + mockDeploymentsList, + mockInference, + parseMockResponse +} from '../../../../test-util/mock-http.js'; +import { OrchestrationClient } from './client.js'; +import type { + CompletionPostResponse, + OrchestrationModuleConfig +} from '@sap-ai-sdk/orchestration'; + +jest.setTimeout(30000); + +describe('orchestration service client', () => { + let mockResponse: CompletionPostResponse; + beforeEach(async () => { + mockClientCredentialsGrantCall(); + mockDeploymentsList({ scenarioId: 'orchestration' }, { id: '1234' }); + mockResponse = await parseMockResponse( + 'orchestration', + 'orchestration-chat-completion-success-response.json' + ); + }); + + afterEach(() => { + nock.cleanAll(); + }); + + function mockInferenceWithResilience(resilience: { + retry?: number; + delay?: number; + }) { + mockInference( + { + data: constructCompletionPostRequest(config, { messagesHistory: [] }) + }, + { + data: mockResponse, + status: 200 + }, + { + url: 'inference/deployments/1234/completion' + }, + resilience + ); + } + + const config: OrchestrationModuleConfig = { + llm: { + model_name: 'gpt-35-turbo-16k', + model_params: { max_tokens: 50, temperature: 0.1 } + }, + templating: { + template: [{ role: 'user', content: 'Hello!' }] + } + }; + + it('returns successful response when maxRetries equals retry configuration', async () => { + mockInferenceWithResilience({ retry: 2 }); + + const client = new OrchestrationClient(config, { + maxRetries: 2 + }); + + expect(await client.invoke([])).toMatchSnapshot(); + }); + + it('throws error response when maxRetries is smaller than required retries', async () => { + mockInferenceWithResilience({ retry: 2 }); + + const client = new OrchestrationClient(config, { + maxRetries: 1 + }); + + await expect(client.invoke([])).rejects.toThrowErrorMatchingInlineSnapshot( + '"Request failed with status code 500"' + ); + }); + + it('throws when delay exceeds timeout', async () => { + mockInferenceWithResilience({ delay: 2000 }); + + const client = new OrchestrationClient(config); + + const response = client.invoke([], { timeout: 1000 }); + + await expect(response).rejects.toThrow( + expect.objectContaining({ + stack: expect.stringMatching(/Timeout/) + }) + ); + }); + + it('returns successful response when timeout is bigger than delay', async () => { + mockInferenceWithResilience({ delay: 2000 }); + + const client = new OrchestrationClient(config); + + const response = await client.invoke([], { timeout: 3000 }); + expect(response).toMatchSnapshot(); + }); +}); diff --git a/packages/langchain/src/orchestration/client.ts b/packages/langchain/src/orchestration/client.ts new file mode 100644 index 000000000..81ec87fdb --- /dev/null +++ b/packages/langchain/src/orchestration/client.ts @@ -0,0 +1,131 @@ +import { BaseChatModel } from '@langchain/core/language_models/chat_models'; +import { OrchestrationClient as OrchestrationClientBase } from '@sap-ai-sdk/orchestration'; +import { + isTemplate, + mapLangchainMessagesToOrchestrationMessages, + mapOutputToChatResult +} from './util.js'; +import type { BaseLanguageModelInput } from '@langchain/core/language_models/base'; +import type { Runnable, RunnableLike } from '@langchain/core/runnables'; +import type { OrchestrationMessageChunk } from './orchestration-message-chunk.js'; +import type { ChatResult } from '@langchain/core/outputs'; +import type { OrchestrationModuleConfig } from '@sap-ai-sdk/orchestration'; +import type { BaseChatModelParams } from '@langchain/core/language_models/chat_models'; +import type { ResourceGroupConfig } from '@sap-ai-sdk/ai-api'; +import type { BaseMessage } from '@langchain/core/messages'; +import type { CallbackManagerForLLMRun } from '@langchain/core/callbacks/manager'; +import type { OrchestrationCallOptions } from './types.js'; +import type { HttpDestinationOrFetchOptions } from '@sap-cloud-sdk/connectivity'; + +/** + * The Orchestration client. + */ +export class OrchestrationClient extends BaseChatModel< + OrchestrationCallOptions, + OrchestrationMessageChunk +> { + constructor( + // TODO: Omit streaming until supported + public orchestrationConfig: Omit, + public langchainOptions: BaseChatModelParams = {}, + public deploymentConfig?: ResourceGroupConfig, + public destination?: HttpDestinationOrFetchOptions + ) { + super(langchainOptions); + } + + _llmType(): string { + return 'orchestration'; + } + + /** + * Create a new runnable sequence that runs each individual runnable in series, + * piping the output of one runnable into another runnable or runnable-like. + * @param coerceable - A runnable, function, or object whose values are functions or runnables. + * @returns A new runnable sequence. + */ + override pipe( + coerceable: RunnableLike + ): Runnable< + BaseLanguageModelInput, + Exclude, + OrchestrationCallOptions + > { + return super.pipe(coerceable) as Runnable< + BaseLanguageModelInput, + Exclude, + OrchestrationCallOptions + >; + } + + override async _generate( + messages: BaseMessage[], + options: typeof this.ParsedCallOptions, + runManager?: CallbackManagerForLLMRun + ): Promise { + const res = await this.caller.callWithOptions( + { + signal: options.signal + }, + () => { + const { inputParams, customRequestConfig } = options; + const mergedOrchestrationConfig = + this.mergeOrchestrationConfig(options); + const orchestrationClient = new OrchestrationClientBase( + mergedOrchestrationConfig, + this.deploymentConfig, + this.destination + ); + const messagesHistory = + mapLangchainMessagesToOrchestrationMessages(messages); + return orchestrationClient.chatCompletion( + { + messagesHistory, + inputParams + }, + customRequestConfig + ); + } + ); + + const content = res.getContent(); + + // TODO: Add streaming as soon as we support it + await runManager?.handleLLMNewToken( + typeof content === 'string' ? content : '' + ); + + return mapOutputToChatResult(res.data); + } + + private mergeOrchestrationConfig( + options: typeof this.ParsedCallOptions + ): OrchestrationModuleConfig { + const { tools = [], stop = [] } = options; + return { + ...this.orchestrationConfig, + llm: { + ...this.orchestrationConfig.llm, + model_params: { + ...this.orchestrationConfig.llm.model_params, + ...(stop.length && { + stop: [ + ...(this.orchestrationConfig.llm.model_params?.stop || []), + ...stop + ] + }) + } + }, + templating: { + ...this.orchestrationConfig.templating, + ...(isTemplate(this.orchestrationConfig.templating) && + tools.length && { + tools: [ + ...(this.orchestrationConfig.templating.tools || []), + ...tools + ] + }) + } + }; + } +} diff --git a/packages/langchain/src/orchestration/index.ts b/packages/langchain/src/orchestration/index.ts new file mode 100644 index 000000000..fac385586 --- /dev/null +++ b/packages/langchain/src/orchestration/index.ts @@ -0,0 +1,5 @@ +export * from './client.js'; +export * from './orchestration-message.js'; +export * from './orchestration-message-chunk.js'; +export * from './types.js'; +export * from './util.js'; diff --git a/packages/langchain/src/orchestration/orchestration-message-chunk.ts b/packages/langchain/src/orchestration/orchestration-message-chunk.ts new file mode 100644 index 000000000..67236b7e7 --- /dev/null +++ b/packages/langchain/src/orchestration/orchestration-message-chunk.ts @@ -0,0 +1,21 @@ +import { AIMessageChunk } from '@langchain/core/messages'; +import type { AIMessageChunkFields } from '@langchain/core/messages'; +import type { ModuleResults } from '@sap-ai-sdk/orchestration'; + +/** + * An AI Message Chunk containing module results and request ID. + * @internal + */ +export class OrchestrationMessageChunk extends AIMessageChunk { + module_results: ModuleResults; + request_id: string; + constructor( + fields: string | AIMessageChunkFields, + module_results: ModuleResults, + request_id: string + ) { + super(fields); + this.module_results = module_results; + this.request_id = request_id; + } +} diff --git a/packages/langchain/src/orchestration/orchestration-message.ts b/packages/langchain/src/orchestration/orchestration-message.ts new file mode 100644 index 000000000..c84fa0320 --- /dev/null +++ b/packages/langchain/src/orchestration/orchestration-message.ts @@ -0,0 +1,21 @@ +import { AIMessage } from '@langchain/core/messages'; +import type { AIMessageFields } from '@langchain/core/messages'; +import type { ModuleResults } from '@sap-ai-sdk/orchestration'; + +/** + * An AI Message containing module results and request ID. + * @internal + */ +export class OrchestrationMessage extends AIMessage { + module_results: ModuleResults; + request_id: string; + constructor( + fields: string | AIMessageFields, + module_results: ModuleResults, + request_id: string + ) { + super(fields); + this.module_results = module_results; + this.request_id = request_id; + } +} diff --git a/packages/langchain/src/orchestration/types.ts b/packages/langchain/src/orchestration/types.ts new file mode 100644 index 000000000..09f4f3806 --- /dev/null +++ b/packages/langchain/src/orchestration/types.ts @@ -0,0 +1,22 @@ +import type { Prompt, Template } from '@sap-ai-sdk/orchestration'; +import type { BaseChatModelCallOptions } from '@langchain/core/language_models/chat_models'; +import type { CustomRequestConfig } from '@sap-ai-sdk/core'; + +/** + * Options for an orchestration call. + */ +export type OrchestrationCallOptions = Pick< + BaseChatModelCallOptions, + | 'stop' + | 'signal' + | 'timeout' + | 'callbacks' + | 'metadata' + | 'runId' + | 'runName' + | 'tags' +> & { + customRequestConfig?: CustomRequestConfig; + tools?: Template['tools']; + inputParams?: Prompt['inputParams']; +}; diff --git a/packages/langchain/src/orchestration/util.test.ts b/packages/langchain/src/orchestration/util.test.ts new file mode 100644 index 000000000..f871db8d1 --- /dev/null +++ b/packages/langchain/src/orchestration/util.test.ts @@ -0,0 +1,202 @@ +import { + AIMessage, + HumanMessage, + SystemMessage, + ToolMessage +} from '@langchain/core/messages'; +import { + mapLangchainMessagesToOrchestrationMessages, + mapOutputToChatResult +} from './util.js'; +import type { OrchestrationMessage } from './orchestration-message.js'; +import type { + CompletionPostResponse, + ResponseMessageToolCall +} from '@sap-ai-sdk/orchestration'; + +describe('mapLangchainMessagesToOrchestrationMessages', () => { + it('should map an array of LangChain messages to Orchestration messages', () => { + const langchainMessages = [ + new SystemMessage('System message content'), + new HumanMessage('Human message content'), + new AIMessage('AI message content') + ]; + + const result = + mapLangchainMessagesToOrchestrationMessages(langchainMessages); + + expect(result).toEqual([ + { role: 'system', content: 'System message content' }, + { role: 'user', content: 'Human message content' }, + { role: 'assistant', content: 'AI message content' } + ]); + }); + + it('should throw error for unsupported message types', () => { + const langchainMessages = [ + new ToolMessage('Tool message content', 'tool-id') + ]; + + expect(() => + mapLangchainMessagesToOrchestrationMessages(langchainMessages) + ).toThrow('Unsupported message type: tool'); + }); +}); + +describe('mapBaseMessageToChatMessage', () => { + it('should map HumanMessage to ChatMessage with user role', () => { + const humanMessage = new HumanMessage('Human message content'); + + // Since mapBaseMessageToChatMessage is internal, we'll test it through mapLangchainMessagesToOrchestrationMessages + const result = mapLangchainMessagesToOrchestrationMessages([humanMessage]); + + expect(result[0]).toEqual({ + role: 'user', + content: 'Human message content' + }); + }); + + it('should map SystemMessage to ChatMessage with system role', () => { + const systemMessage = new SystemMessage('System message content'); + + const result = mapLangchainMessagesToOrchestrationMessages([systemMessage]); + + expect(result[0]).toEqual({ + role: 'system', + content: 'System message content' + }); + }); + + it('should map AIMessage to ChatMessage with assistant role', () => { + const aiMessage = new AIMessage('AI message content'); + + const result = mapLangchainMessagesToOrchestrationMessages([aiMessage]); + + expect(result[0]).toEqual({ + role: 'assistant', + content: 'AI message content' + }); + }); + + it('should throw error when mapping SystemMessage with image_url content', () => { + const systemMessage = new SystemMessage({ + content: [ + { type: 'text', text: 'System text' }, + { + type: 'image_url', + image_url: { url: 'https://example.com/image.jpg' } + } + ] + }); + + expect(() => + mapLangchainMessagesToOrchestrationMessages([systemMessage]) + ).toThrow( + 'System messages with image URLs are not supported by the Orchestration Client.' + ); + }); +}); + +describe('mapOutputToChatResult', () => { + it('should map CompletionPostResponse to ChatResult', () => { + const completionResponse: CompletionPostResponse = { + orchestration_result: { + id: 'test-id', + object: 'chat.completion', + created: 1634840000, + model: 'test-model', + choices: [ + { + message: { + content: 'Test content', + role: 'assistant' + }, + finish_reason: 'stop', + index: 0 + } + ], + usage: { + completion_tokens: 10, + prompt_tokens: 20, + total_tokens: 30 + } + }, + request_id: 'req-123', + module_results: {} + }; + + const result = mapOutputToChatResult(completionResponse); + + expect(result.generations).toHaveLength(1); + expect(result.generations[0].text).toBe('Test content'); + expect(result.generations[0].message.content).toBe('Test content'); + expect(result.generations[0].generationInfo).toEqual({ + finish_reason: 'stop', + index: 0, + function_call: undefined, + tool_calls: undefined + }); + expect(result.llmOutput).toEqual({ + created: 1634840000, + id: 'test-id', + model: 'test-model', + object: 'chat.completion', + tokenUsage: { + completionTokens: 10, + promptTokens: 20, + totalTokens: 30 + } + }); + }); + + it('should map tool_calls correctly', () => { + const toolCallData: ResponseMessageToolCall = { + id: 'call-123', + type: 'function', + function: { + name: 'test_function', + arguments: '{"arg1":"value1"}' + } + }; + + const completionResponse: CompletionPostResponse = { + orchestration_result: { + id: 'test-id', + object: 'chat.completion', + created: 1634840000, + model: 'test-model', + choices: [ + { + index: 0, + message: { + content: 'Test content', + role: 'assistant', + tool_calls: [toolCallData] + }, + finish_reason: 'tool_calls' + } + ], + usage: { + completion_tokens: 10, + prompt_tokens: 20, + total_tokens: 30 + } + }, + request_id: 'req-123', + module_results: {} + }; + + const result = mapOutputToChatResult(completionResponse); + + expect( + (result.generations[0].message as OrchestrationMessage).tool_calls + ).toEqual([ + { + id: 'call-123', + name: 'test_function', + args: { arg1: 'value1' }, + type: 'tool_call' + } + ]); + }); +}); diff --git a/packages/langchain/src/orchestration/util.ts b/packages/langchain/src/orchestration/util.ts new file mode 100644 index 000000000..2785e5c7c --- /dev/null +++ b/packages/langchain/src/orchestration/util.ts @@ -0,0 +1,179 @@ +import { OrchestrationMessage } from './orchestration-message.js'; +import type { ChatResult } from '@langchain/core/outputs'; +import type { + ChatMessage, + CompletionPostResponse, + Template +} from '@sap-ai-sdk/orchestration'; +import type { ToolCall } from '@langchain/core/messages/tool'; +import type { AzureOpenAiChatCompletionMessageToolCalls } from '@sap-ai-sdk/foundation-models'; +import type { + AIMessage, + BaseMessage, + HumanMessage, + SystemMessage +} from '@langchain/core/messages'; + +/** + * Checks if the object is a {@link Template}. + * @param object - The object to check. + * @returns True if the object is a {@link Template}. + * @internal + */ +export function isTemplate(object: Record): object is Template { + return 'template' in object; +} + +/** + * Maps {@link BaseMessage} to {@link ChatMessage}. + * @param message - The message to map. + * @returns The {@link ChatMessage}. + */ +// TODO: Add mapping of refusal property, once LangChain base class supports it natively. +function mapBaseMessageToChatMessage(message: BaseMessage): ChatMessage { + switch (message.getType()) { + case 'ai': + return mapAiMessageToAzureOpenAiAssistantMessage(message); + case 'human': + return mapHumanMessageToChatMessage(message); + case 'system': + return mapSystemMessageToAzureOpenAiSystemMessage(message); + // TODO: As soon as tool messages are supported by orchestration, create mapping function similar to our azure mapping function. + case 'function': + case 'tool': + default: + throw new Error(`Unsupported message type: ${message.getType()}`); + } +} + +/** + * Maps LangChain's {@link AIMessage} to Azure OpenAI's {@link AzureOpenAiChatCompletionRequestAssistantMessage}. + * @param message - The {@link AIMessage} to map. + * @returns The Azure OpenAI {@link AzureOpenAiChatCompletionRequestAssistantMessage}. + */ +function mapAiMessageToAzureOpenAiAssistantMessage( + message: AIMessage +): ChatMessage { + /* TODO: Tool calls are currently bugged in orchestration, pass these fields as soon as orchestration supports it. + const tool_calls = + mapLangchainToolCallToAzureOpenAiToolCall(message.tool_calls) ?? + message.additional_kwargs.tool_calls; + */ + return { + /* TODO: Tool calls are currently bugged in orchestration, pass these fields as soon as orchestration supports it. + ...(tool_calls?.length ? { tool_calls } : {}), + function_call: message.additional_kwargs.function_call, + */ + content: message.content, + role: 'assistant' + } as ChatMessage; +} + +function mapHumanMessageToChatMessage(message: HumanMessage): ChatMessage { + return { + role: 'user', + content: message.content + } as ChatMessage; +} + +function mapSystemMessageToAzureOpenAiSystemMessage( + message: SystemMessage +): ChatMessage { + // TODO: Remove as soon as image_url is a supported input for system messages in orchestration. + if ( + typeof message.content !== 'string' && + message.content.some(content => content.type === 'image_url') + ) { + throw new Error( + 'System messages with image URLs are not supported by the Orchestration Client.' + ); + } + return { + role: 'system', + content: message.content + } as ChatMessage; +} + +/** + * Maps LangChain messages to orchestration messages. + * @param messages - The LangChain messages to map. + * @returns The orchestration messages mapped from LangChain messages. + * @internal + */ +export function mapLangchainMessagesToOrchestrationMessages( + messages: BaseMessage[] +): ChatMessage[] { + return messages.map(mapBaseMessageToChatMessage); +} + +/** + * Maps {@link AzureOpenAiChatCompletionMessageToolCalls} to LangChain's {@link ToolCall}. + * @param toolCalls - The {@link AzureOpenAiChatCompletionMessageToolCalls} response. + * @returns The LangChain {@link ToolCall}. + */ +function mapAzureOpenAiToLangchainToolCall( + toolCalls?: AzureOpenAiChatCompletionMessageToolCalls +): ToolCall[] | undefined { + if (toolCalls) { + return toolCalls.map(toolCall => ({ + id: toolCall.id, + name: toolCall.function.name, + args: JSON.parse(toolCall.function.arguments), + type: 'tool_call' + })); + } +} + +/** + * Maps the completion response to a {@link ChatResult}. + * @param completionResponse - The completion response to map. + * @returns The mapped {@link ChatResult}. + * @internal + */ +export function mapOutputToChatResult( + completionResponse: CompletionPostResponse +): ChatResult { + const { orchestration_result, module_results, request_id } = + completionResponse; + const { choices, created, id, model, object, usage, system_fingerprint } = + orchestration_result; + return { + generations: choices.map(choice => ({ + text: choice.message.content ?? '', + message: new OrchestrationMessage( + { + content: choice.message.content ?? '', + tool_calls: mapAzureOpenAiToLangchainToolCall( + choice.message.tool_calls + ), + additional_kwargs: { + finish_reason: choice.finish_reason, + index: choice.index, + function_call: choice.message.function_call, + tool_calls: choice.message.tool_calls + } + }, + module_results, + request_id + ), + generationInfo: { + finish_reason: choice.finish_reason, + index: choice.index, + function_call: choice.message.function_call, + tool_calls: choice.message.tool_calls + } + })), + llmOutput: { + created, + id, + model, + object, + system_fingerprint, + tokenUsage: { + completionTokens: usage?.completion_tokens ?? 0, + promptTokens: usage?.prompt_tokens ?? 0, + totalTokens: usage?.total_tokens ?? 0 + } + } + }; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 314ea632a..78561e492 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -157,6 +157,9 @@ importers: '@sap-ai-sdk/foundation-models': specifier: workspace:^ version: link:../foundation-models + '@sap-ai-sdk/orchestration': + specifier: workspace:^ + version: link:../orchestration '@sap-cloud-sdk/connectivity': specifier: ^3.26.1 version: 3.26.1 @@ -6186,7 +6189,7 @@ snapshots: eslint: 9.21.0 eslint-config-prettier: 10.0.1(eslint@9.21.0) eslint-import-resolver-typescript: 3.8.1(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint@9.21.0))(eslint@9.21.0) - eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint-import-resolver-typescript@3.8.1)(eslint@9.21.0) + eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint-import-resolver-typescript@3.8.1(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint@9.21.0))(eslint@9.21.0))(eslint@9.21.0) eslint-plugin-jsdoc: 50.6.3(eslint@9.21.0) eslint-plugin-prettier: 5.2.3(@types/eslint@8.56.10)(eslint-config-prettier@10.0.1(eslint@9.21.0))(eslint@9.21.0)(prettier@3.5.2) eslint-plugin-regex: 1.10.0(eslint@9.21.0) @@ -7626,7 +7629,7 @@ snapshots: stable-hash: 0.0.4 tinyglobby: 0.2.11 optionalDependencies: - eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint-import-resolver-typescript@3.8.1)(eslint@9.21.0) + eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint-import-resolver-typescript@3.8.1(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint@9.21.0))(eslint@9.21.0))(eslint@9.21.0) transitivePeerDependencies: - supports-color @@ -7641,7 +7644,7 @@ snapshots: transitivePeerDependencies: - supports-color - eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint-import-resolver-typescript@3.8.1)(eslint@9.21.0): + eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint-import-resolver-typescript@3.8.1(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.24.1(eslint@9.21.0)(typescript@5.7.3))(eslint@9.21.0))(eslint@9.21.0))(eslint@9.21.0): dependencies: '@rtsao/scc': 1.1.0 array-includes: 3.1.8 diff --git a/sample-code/README.md b/sample-code/README.md index 6188bb66b..88705c59d 100644 --- a/sample-code/README.md +++ b/sample-code/README.md @@ -204,22 +204,29 @@ The `toContentStream()` method is called to extract the content of the chunk for Once the streaming is done, finish reason and token usage are printed out. -### Langchain +### LangChain #### Invoke with a Simple Input `GET /langchain/invoke` -Invoke langchain Azure OpenAI client with a simple input to get chat completion response. +Invoke LangChain Azure OpenAI client with a simple input to get chat completion response. -#### Invoke a Chain for Templating +#### Invoke a Chain with Templating `GET /langchain/invoke-chain` Invoke chain to get chat completion response from Azure OpenAI. The chain contains a template and a string parser. -#### Invoke a Chain for Retrieval-Augmented Generation (RAG) +#### Invoke a Chain with Templating and Orchestration Client + +`GET /langchain/invoke-chain-orchestration` + +Invoke a chain to get a orchestration response from the orchestration service. +The chain has a built-in template and is chained with a string parser. + +#### Invoke a Chain with Retrieval-Augmented Generation (RAG) `GET /langchain/invoke-rag-chain` diff --git a/sample-code/src/index.ts b/sample-code/src/index.ts index a6a5938e0..609157de4 100644 --- a/sample-code/src/index.ts +++ b/sample-code/src/index.ts @@ -28,6 +28,7 @@ export { invokeChain, invokeRagChain } from './langchain-azure-openai.js'; +export { invokeChain as orchestrationInvokeChain } from './langchain-orchestration.js'; export { getDeployments, getDeploymentsWithDestination, diff --git a/sample-code/src/langchain-orchestration.ts b/sample-code/src/langchain-orchestration.ts new file mode 100644 index 000000000..970bc7610 --- /dev/null +++ b/sample-code/src/langchain-orchestration.ts @@ -0,0 +1,39 @@ +import { StringOutputParser } from '@langchain/core/output_parsers'; +import { OrchestrationClient } from '@sap-ai-sdk/langchain'; + +/** + * ASk GPT about an introduction to SAP Cloud SDK. + * @returns The answer from ChatGPT. + */ +export async function invokeChain(): Promise { + const orchestrationConfig = { + // define the language model to be used + llm: { + model_name: 'gpt-35-turbo', + model_params: {} + }, + // define the template + templating: { + template: [ + { + role: 'user', + content: 'Give me a long introduction of {{?input}}' + } + ] + } + }; + + const callOptions = { inputParams: { input: 'SAP Cloud SDK' } }; + + // initialize the client + const client = new OrchestrationClient(orchestrationConfig); + + // create an output parser + const parser = new StringOutputParser(); + + // chain together template, client, and parser + const llmChain = client.pipe(parser); + + // invoke the chain + return llmChain.invoke('My Message History', callOptions); +} diff --git a/sample-code/src/server.ts b/sample-code/src/server.ts index 0c5e3f5f8..c81aae100 100644 --- a/sample-code/src/server.ts +++ b/sample-code/src/server.ts @@ -42,6 +42,7 @@ import { invoke, invokeToolChain } from './langchain-azure-openai.js'; +import { invokeChain as invokeChainOrchestration } from './langchain-orchestration.js'; import { createCollection, createDocumentsWithTimestamp, @@ -416,6 +417,17 @@ app.get('/langchain/invoke-chain', async (req, res) => { } }); +app.get('/langchain/invoke-chain-orchestration', async (req, res) => { + try { + res.send(await invokeChainOrchestration()); + } catch (error: any) { + console.error(error); + res + .status(500) + .send('Yikes, vibes are off apparently 😬 -> ' + error.request.data); + } +}); + app.get('/langchain/invoke-rag-chain', async (req, res) => { try { res.send(await invokeRagChain()); diff --git a/test-util/mock-http.ts b/test-util/mock-http.ts index ff01e4a73..77409057e 100644 --- a/test-util/mock-http.ts +++ b/test-util/mock-http.ts @@ -2,18 +2,18 @@ import { readFile } from 'node:fs/promises'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import nock from 'nock'; -import { type EndpointOptions } from '@sap-ai-sdk/core'; -import { - type FoundationModel, - type DeploymentResolutionOptions -} from '@sap-ai-sdk/ai-api/internal.js'; -import { dummyToken } from './mock-jwt.js'; import { registerDestination, type DestinationAuthToken, type HttpDestination, type ServiceCredentials } from '@sap-cloud-sdk/connectivity'; +import { type EndpointOptions } from '@sap-ai-sdk/core'; +import { + type FoundationModel, + type DeploymentResolutionOptions +} from '@sap-ai-sdk/ai-api/internal.js'; +import { dummyToken } from './mock-jwt.js'; // Get the directory of this file const __filename = fileURLToPath(import.meta.url); @@ -98,19 +98,37 @@ export function mockInference( data: any; status?: number; }, - endpoint: EndpointOptions = mockEndpoint + endpoint: EndpointOptions = mockEndpoint, + resilienceOptions?: { + delay?: number; + retry?: number; + } ): nock.Scope { const { url, apiVersion, resourceGroup = 'default' } = endpoint; const destination = getMockedAiCoreDestination(); - return nock(destination.url, { + const scope = nock(destination.url, { reqheaders: { 'ai-resource-group': resourceGroup, authorization: `Bearer ${destination.authTokens?.[0].value}` } - }) - .post(`/v2/${url}`, request.data) - .query(apiVersion ? { 'api-version': apiVersion } : {}) - .reply(response.status, response.data); + }); + + let interceptor = scope.post(`/v2/${url}`, request.data).query(apiVersion ? { 'api-version': apiVersion } : {}); + + if (resilienceOptions?.retry) { + interceptor = interceptor.times(resilienceOptions.retry); + if(resilienceOptions.delay) { + interceptor = interceptor.delay(resilienceOptions.delay); + } + interceptor.reply(500); + interceptor = scope.post(`/v2/${url}`, request.data).query(apiVersion ? { 'api-version': apiVersion } : {}); + } + + if (!resilienceOptions?.retry && resilienceOptions?.delay) { + interceptor = interceptor.delay(resilienceOptions.delay); + } + + return interceptor.reply(response.status, response.data); } /** diff --git a/tests/e2e-tests/src/orchestration-langchain.test.ts b/tests/e2e-tests/src/orchestration-langchain.test.ts new file mode 100644 index 000000000..7c39ac38e --- /dev/null +++ b/tests/e2e-tests/src/orchestration-langchain.test.ts @@ -0,0 +1,11 @@ +import { orchestrationInvokeChain } from '@sap-ai-sdk/sample-code'; +import { loadEnv } from './utils/load-env.js'; + +loadEnv(); + +describe('Orchestration LangChain client', () => { + it('executes invoke as part of a chain ', async () => { + const result = await orchestrationInvokeChain(); + expect(result).toContain('SAP Cloud SDK'); + }); +});