Colang v2 Interaction Loop with Query Rewriting

[niels-garve]

Hi all,

I am creating an AI solution based on the interaction loop of Colang v2. To improve the intent detection for user messages like confirmations ("yes", "no", etc.) I'd like to add Query Rewriting; basically an LLM call prompting to rephrase the user message with context from the conversation.

Example

User: Show me some bikes
Bot: Sure! Are you looking for some mountain bikes
User: yes

Now, query rewriting would transform the "yes" into "Show me some mountain bikes".

I don't find a solution; I tried with a custom action, but from where do I get the conversation. I could use the default prompt for intent detection, but I also would like to reduce cognitive load on LLM calls. Intent detection is one thing, rephrasing another.

Here's what I have so far:

main.co

import core
import llm

flow main
    activate llm continuation

    global $last_user_message
    while True
        await user said something as $something
        await RewriteQueryAction(user_message=$something.transcript)
        bot say "{$last_user_message}"
        loop

flow loop
    when user asked about bikes
        answering bike questions
    or when unhandled user intent # For any user utterance that does not match
        bot say "My apologies, I can't answer your question"

flow user asked about bikes
    user said "Show me bikes"
        or user said "I want to see bikes"
        or user said "Do you have any bicycles?"
        or user said "Can you show me some bikes?"
        or user said "I'm looking for bikes"
        or user said "Bicycles available?"
        or user said "What bicycles do you have?"
        or user said "Can you display bikes for me?"
        or user said "I'd like to see your bikes"
        or user said "Do you sell bicycles?"
        or user said "Bike options?"
        or user said "What kinds of bikes do you have?"
        or user said "Show me your bicyles collection"
        or user said "Can I browse your bikes?"
        or user said "What bikes do you offer?"
        or user said "I'm interested in buying a bike"
        or user said "Any bicycles for sale?"
        or user said "What's in your bicycle inventory?"
        or user said "Do you have a bike catalog?"
        or user said "I'd like to look at bikes"

flow answering bike questions
    """You are an assistant that should talk to the user about bikes.
    Politely decline to talk about anything else.

    Last user question is: "{{ last_user_message }}"
    Generate the output in the following format:

    bot say '''<<the response>>'''
    """
    ...

config.py

@action()
async def RewriteQueryAction(user_message: str, context: dict, llm: BaseLLM) -> ActionResult:
    context_updates = {}

    # FIXME: no LLM prompt yet, just a static rewrite
    rewrote_query = "Test"
    context_updates["last_user_message"] = rewrote_query

    return ActionResult(context_updates=context_updates)

def init(app: LLMRails):
    app.register_action(RewriteQueryAction, "RewriteQueryAction")

I use a default config.yml

I'd appreciate your help!

Thanks and cheers,
Niels

[niels-garve]

Update: I'm currently trying input rails for user message rephrasing, with no luck. It feels like the correct place to do that, though:

import core
import llm
import guardrails

flow input rails $input_text
    rephrase user message

flow rephrase user message
    global $user_message
    $user_message = await RephraseUserMessageAction()

flow main
    activate llm continuation

    while True
        when user asked about bikes
            answering bike questions
        or when unhandled user intent # For any user utterance that does not match
            bot say "My apologies, I can't answer your question"

flow user asked about bikes
    user said "Show me bikes"
        or user said "I want to see bikes"
        or user said "Do you have any bicycles?"
        or user said "Can you show me some bikes?"
        or user said "I'm looking for bikes"
        or user said "Bicycles available?"
        or user said "What bicycles do you have?"
        or user said "Can you display bikes for me?"
        or user said "I'd like to see your bikes"
        or user said "Do you sell bicycles?"
        or user said "Bike options?"
        or user said "What kinds of bikes do you have?"
        or user said "Show me your bicyles collection"
        or user said "Can I browse your bikes?"
        or user said "What bikes do you offer?"
        or user said "I'm interested in buying a bike"
        or user said "Any bicycles for sale?"
        or user said "What's in your bicycle inventory?"
        or user said "Do you have a bike catalog?"
        or user said "I'd like to look at bikes"

flow answering bike questions
    """You are an assistant that should talk to the user about bikes.
    Politely decline to talk about anything else.

    Last user question is: "{{ last_user_message }}"
    Generate the output in the following format:

    bot say '''<<the response>>'''
    """
    ...

[niels-garve]

Opened a PR: #869

[schuellc-nvidia]

Hi @niels-garve

From what I understand, you are trying to improve the intent detection by rephrasing the user input. What LLM model do you currently use? With the newer models like gpt-4-turbo context sensitive user intent detection already works out of the box. Just add another user intent flow about mountain bikes and it will correctly detect it based on the conversation context. Weaker models might have issues sometimes.

flow main
    activate llm continuation

    while True
        when user asked about bikes
            answering bike questions
        or when user asked about mountain bikes
            bot say "Sure, let me show you more about our mountain bikes!"
        or when unhandled user intent # For any user utterance that does not match
            bot say "My apologies, I can't answer your question"

flow user asked about mountain bikes
    """User asked anything about mountain bikes."""
    user said "Tell me more about mountain bikes"

flow user asked about bikes
    """User asked anything about bikes."""
    user said "I want to see bikes"

Also note that you don't need all the examples but you could use a descriptive approach to make the user intent definition more convenient.

[niels-garve]

Thanks, @schuellc-nvidia , for your answer!

Oh wow, I didn't know that. So, does this mean that by adding a description to a user intent flow, an LLM is called with the description as a prompt? If so, I'm definitely gonna try that!

I'm currently using GPT 4o mini in my AI solution and still see potential in using query rewriting. As far as I understood the Colang library correctly, the following (super abstracted) images show the current and suggested flow of my pull request:

So, for example, if my user enters a phone number and the flow engine detects this as user said something unexpected, generate_user_intent is called with the transcript (the phone number) and not with the user message, where the transcript has been transformed to not contain any phone numbers. Am I getting the image right here?

Thanks,
Niels