Dynamic config.json

[mpriem]

Hi,

We would like to dynamically set the model parameters that are in config.json. (Basically by loading them from an external source so data scientists can tweak without engineering having to redeploy)

Is there an easy way of doing that?

Thanks!

[corinagum]

Hi @mpriem, as far as I know there isn't a way to dynamically set parameters in the way you are saying. I have a few suggestions but none of them are an exact match to your ask:

1. Instead of using the config.json, there is the option to make changes via typescript, like in this code example. Recommending it on overall being more dynamic, but not a solution.

prompt.config.completion = {
    "temperature": process.env.TEMPERATURE, 
     // other params here...
}

This is a way to set the options as environment keys loading from an external file. However, this would still need a deployment for that file that's being updated.

2. Get a locally running app for the data scientists to fiddle with, since they would only be touching one file of the codebase and don't need to make pushes. You'd still be making calls to the LLM but it removes the layer of needing to push changes. This is good since you probably don't want to be making tweaks on prod. Our samples use Teams Toolkit, which make it super easy for non-coders to get the app up and running once everything has been set up.
3. Use the LLM playground instead -- Azure OpenAI playground in particular I find pretty convenient to use, and that playground and the OpenAI playground are constantly evolving. This is a no-code solution but does remove features specific to your codebase. The plus side of this is that they would be testing strictly the LLM and can test without worrying about the code.

Hope that helps!

[mpriem]

Hi @corinagum ,

Thanks for the reply. Unfortunately this won't work in our case as the bot and model is part of a much larger solution so not easily isolated.

I was looking at the prompt manager interface. Would it be possible to create prompts dynamically?
I haven't tried yet and I also do not understand how with multiple prompts the correct one gets picked or if an how to force the use of a specific prompt. Would that be an option?

Thanks

[corinagum]

The planner defaultPrompt can be a function and therefore dynamically set the config like so:
https://github.com/microsoft/teams-ai/blob/a441b82ad71b502a1d0c451f9ea45c89eb8ed339/js/samples/04.ai-apps/h.datasource-azureOpenAI/src/app.ts#L38-L58

[mpriem]

Ok, so that would mean we would only have to restart the app to have the config update. That is definitely better than needing to redeploy.

Thanks for your help.

[heyitsaamir]

We're clearing discussions and starting fresh. This discussion has not been acted upon. Please reopen the discussion, or start a new one if this question is still important. Thank you for your understanding.