How to handle widget actions in Agent Builder workflows?

[ziggy6792]

Hey team

I’m trying to better understand widget actions in the Agent Builder.

For example, I’ve added a custom widget with some buttons in my Agent Builder workflow:

When I click one of these buttons, I see a network request going out to my Agent Builder workflow endpoint:

curl 'https://api.openai.com/v1/chatkit/conversation' \
  --data-raw '{
    "type":"threads.custom_action",
    "params":{
      "item_id":"cti_68f26792110c8190b5beda4db8a1ca860836fedd48df3569",
      "action":{
        "type":"session.save",
        "payload":{"id":"SESSION_ID"}
      },
      "thread_id":"cthr_68f2677d38d8819089b6a09716cdfcbc0836fedd48df3569"
    }
  }'

However, it seems like Agent Builder workflows (as in workflows build in Agent Builder ui) currently don’t have a way to handle these actions, right?

---

I also tried handling this on the frontend using the widgets.onAction callback:

const chatkit = useChatKit({
  ...
  widgets: {
    onAction: async (
      action: { type: string; payload?: Record<string, unknown> },
      widgetItem: { id: string; widget: Widgets.Card | Widgets.ListView }
    ) => {
      console.log('action', action);
      console.log('widgetItem', widgetItem);
    },
  },
});

But this function doesn’t seem to get triggered when I click the button either.

---

Question:

Am I correct that the only way to handle widget actions right now is by writing a custom workflow server, similar to what’s shown here?
openai/openai-chatkit-advanced-samples

Thanks for the clarification!

[crtag]

@ziggy6792
These are React "actions" you meant to deal with in your front-end application, intended for custom implementation, as your components technically can be anything with their own APIs.

[vanha777]

i got a tool name RepGuy, which can serve your workflows within your mobile, now i've implemented trigger widgets function on native mobile device like call send sms and add contact ......
https://repguy.dev/docs

if u interested, let's work together

[hieuminh65]

const chatkit = useChatKit({
  ...
  widgets: {
    onAction: async (
      action: { type: string; payload?: Record<string, unknown> },
      widgetItem: { id: string; widget: Widgets.Card | Widgets.ListView }
    ) => {
      console.log('action', action);
      console.log('widgetItem', widgetItem);
    },
  },
});

I tried this as well, did you see any log from here? I try to click the button in the widget, which has its correct action config like the docs but nothing is triggered

[HunterHillegas]

This works for me... In my widget I have something like this.

Make sure you have handler: client set.

      <Button
        label="Open"
        variant="outline"
        onClickAction={{
          type: "video.open",
          payload: { site_url: item.site_url },
          handler: "client"
        }}
      />

And then in my ChatKit JS code:

widgets: {
  async onAction(action, item) {
    if (action.type === "video.open") {
      // do stuff
    } else {
      console.log(`Action Not Handled: ${action} | ${item}`);
    }
  },
},