Allow ai.action as well as app.activity(ActivityTypes.Message....) [TS]

[KriyenKP]

I've been using the typescript libraries and have been trying to incorporate actions as well as conversation via a.i.

My intent is to have natural conversation like chatgpt but when an action is found, then perform that. However I've found that when I use the catch all "app.activity(ActivityTypes.Message,..." the "app.ai.action('lightsOn..." gets ignored completely.

Can anyone provide some insight? I'll post code if required.

[corinagum]

My apologies for the late reply.

It is expected that app.activity(ActivityTypes.Message... would override other handlers as it takes precedence in terms of 'activity coverage'. Without having seen your code, one thing you could try is make sure that the app.ai.action code is ABOVE the app.activity code to check whether or not it will be caught before the catch-all. This is because routes within the app are processed in order that they are listed in the code.

Otherwise, if that doesn't work, it would be best if you found a way to narrow down scope for app.activity so it doesn't act as a catch-all.

[KriyenKP]

@corinagum
An example to minimize company IP issues :

app.ai.action('generate', async (context: TurnContext, state: ApplicationTurnState) => {
//Generate image snippet
});

app.activity(ActivityTypes.Message, async (context: TurnContext, state: ApplicationTurnState) => {
// if they haven't asked for image generation then just have normal conversation/responses based on GPT model
});

[corinagum]

the catch-all here is redundant because your app.ai. routes will catch action behavior and route responses through GPT anyway. Without further information on what you're doing inside of app.activity I can't help you narrow down scope further.

[KriyenKP]

When I don't include the app.activity(ActivityTypes.Message, the bot responds with something to the effect of "this bot only knows how to generate an image". The only reason I've added it in is to try and send it to another bot that only handles conversation. There isn't any other intended functionality.

If someone says "generate an image based of a dog" it should do so by using an included function containing an api call and response.
If someone says "give me a recipe for pancakes" it should use the GPT model to do so.

The approach I was forced to take was to use app.activity("/generateImage"... (note the slash in the string) and app.activity(ActivityTypes.Message, to cater for an action and then to handles any other messages.

[corinagum]

1. I would recommend trying out regex instead of a string for generate image text - that way, the user isn't expected to follow a super strict structure to do a specific task.
2. Generally, the developer's goal is to limit the scope of interaction with AI so that tokens/credits aren't spent on conversation that the AI isn't meant to facilitate. If you know some of the topics that the customer may bring up, like pancakes, I would say that would justify having their own prompt.
3. As for a catch-all for messages, you might experiment with a general prompt called 'chat' or something that tells the AI to handle other conversations. Again, app.activity(ActivityTypes.Message... is not the solution you're looking for.

I'll speak with my team to see if there are other suggestions for a catch-all for conversation.

[corinagum]

@KriyenKP to add on, instead of using app.activity, I think you want to use app.message(/regex for generage images here/...

[lifan0127]

@KriyenKP, yes, I just confirmed with @Stevenic that a route like app.activity.(ActivityTypes.Message... will be a catch-all and interfere with other routes.

Could you share your code for what you're trying to do in that code snippet? Depending on what you're trying to do, there may be better alternatives to either narrow down scope or use beforeTurn and/or afterTurn to accomplish your goals.

Hello @corinagum , in my tests, app.activity.(ActivityTypes.Message..., even placed at the very end, will interfere with app.ai.actions... routes. Is that the intended behavior?

[corinagum]

As what you quoted says, app.activity is a catch-all that interferes with other routes. It should not be used. Please look at the other guidance I have posted above and test those.

[lifan0127]

@corinagum Thanks for confirming. I roughly understand why app.activity will override app.ai.actions based on the implementation, but I think this behavior is different from other "routes" such as app.messageExtensions.*, app.adaptiveCards.* etc. Such inconsistency may be confusing to developers.

I understand that we can set up different handlers based on message pattern matching, as shown in your examples. In my case, I need to have different handlers based on the conversation state, not text patterns. How would I be able to do it without using app.activity?

Thanks!

[corinagum]

app.activity will intercept all activities. Route selectors are iterated through in order, which is why they should be last.

The route selectors don't inherently contain turn state, just context.
beforeTurn has access to turn context and state, so you could potentially check if there was a change of state and add it into the context.

    context.activity.value = {
        settingName: value
    }

[lifan0127]

Thank you, @corinagum. I realize that there are still a lot for me to learn and it would be very helpful to have more documentation and advanced examples.