Streaming LLM responses

When the LLM node has Stream enabled, it does not wait for the full response. Instead, it immediately returns a streamKey and begins accumulating the response in the background. You can then read a snapshot of the response so far using the Read LLM Stream node.

This allows you to process partial output while the model is still generating.

Creating a streaming behavior

We create a simple behavior that sends messages to the user as they arrive from the LLM stream. We treat each paragraph in the LLM response as a separate message.

You can view the completed behavior at openrp/behavior-examples.

Step 1: Set up

Start by setting up the same foundation from the Getting Started guide: receive a chat message, load the chat data, and identify which character should respond.

1. Add a Chat Message event node (chatMessage).

2. Connect a Get Chat node (getChat).

   • Chat ID: chatMessage.chatId
   • Expand Relations: select participants and messages.

3. Connect a Filter node (filter).

   • List: getChat.participants.data
   • Item Condition: item.userId === null

   We use this to keep only the AI characters in the chat.

4. Connect an LLM node (llm).

   • Prompt: {{getChat.messages.data[0].content}}
   • Model ID: chatMessage.modelSettings.chatModelId
   • Stream: true

   Since Stream is enabled, the LLM node returns immediately with a streamKey instead of blocking the execution.

5. Connect a Set Variable node (setVariable).

   • Variables:
     • Key: paragraphsSent
     • Value: 0

   We will use this variable in Step 3, to track which paragraph we have sent so far.

Step 2: Poll the stream

We now create a loop that repeatedly reads the LLM stream until it is complete.

1. From setVariable, connect a Repeat Until node (repeatUntil).

   • Expression: !readLlmStream.isFinished

   This is the loop that will repeatedly read the LLM stream until it is complete. Note that we haven't added readLlmStream node yet, so it won't show up in auto-complete.

2. From the loopStart output of repeatUntil, connect a Wait node (wait).

   This adds a delay between polls. Without it, the loop would spin as fast as possible without giving the stream time to accumulate more text.

3. Connect a Read LLM Stream node (readLlmStream).

   • Stream Key: llm.streamKey

Step 3: Insert messages

We want to send messages to the chat as soon as they arrive. Let's break up the response into separate paragraphs, and send each one as a separate message.

To avoid sending the same message twice, we use the variable we defined in Step 1 to keep track of which paragraphs we have already sent.

1. From readLlmStream, connect a String Split node (stringSplit).

   • Text: readLlmStream.snapshot.choices[0].message.content || ""

   This breaks down the LLM response into separate paragraphs, using the default double newline as the separator.

   Note

   The || "" handles early iterations where the snapshot may not have any content yet, in which case text would evaluate to an empty string "".

2. Connect an If node (if).

   • Expression: (stringSplit.array.length - (readLlmStream.isFinished ? 0 : 1)) > $variables.paragraphsSent

   This expression checks whether the split array has more paragraphs than we've already sent. While the stream is still running, we skip the last segment (hence we minus 1) because it's incomplete - AI is still writing that paragraph. Once the stream finishes, we include it.

3. From the True output of if, connect a Repeat Until node (repeatUntil2).

   • Expression: same as the If: (stringSplit.array.length - (readLlmStream.isFinished ? 0 : 1)) > $variables.paragraphsSent

   We will send all unsent paragraphs to the chat with this inner loop.

4. From the loopStart of repeatUntil2, connect an Insert Chat Message node (insertChatMessage).

   • Chat ID: getChat.id
   • Content: {{stringSplit.array[$variables.paragraphsSent]}}
   • Chat Participant ID: filter.list[0].id

   This sends the unsent paragraph as a chat message from the AI character.

5. Connect a Set Variable node (setVariable).

   • Variables:
     • Key: paragraphsSent
     • Value: $variables.paragraphsSent + 1

6. Connect a Wait node (wait2).

7. Connect wait2 back to the loopEnd input of repeatUntil2.

8. From the next output of repeatUntil2, connect an End If node (endIf).

9. From the False output of if, connect to the same endIf node.

10. Connect endIf to the loopEnd input of repeatUntil (the outer loop).

Testing the behavior

If you did everything correctly, you should end up with something similar to openrp/behavior-examples.

Now you can follow the test your behavior guide to run your behavior.

Wrapping up

In this guide, we created a behavior that uses two loops to stream an LLM response as it is being generated. The outer loop polls the stream for the current snapshot, while the inner loop detects whether there are any new paragraphs and sends them to the chat.

Depending on your use case, you may extend this example to do other things, such as asking the LLM to output separate messages for separate chat participants, or stop generation early if we detect someone else sent a message to the chat with a Get Chat Messages node.