docs: add human-in-the-loop guides for OpenAI and Anthropic by GregHolmes · Pull Request #3133 · ably/docs

GregHolmes · 2026-01-21T11:20:28Z

Add two new AI Transport guides demonstrating how to implement human-in-the-loop approval workflows for AI agent tool calls:

openai-human-in-the-loop.mdx: HITL with OpenAI function calling
anthropic-human-in-the-loop.mdx: HITL with Anthropic tool use

Both guides cover:

Defining tools requiring human approval
Publishing approval requests with requestId correlation
Subscribing to and processing approval decisions
Progress streaming during long-running tool execution
Handling rejection gracefully

coderabbitai · 2026-01-21T11:20:38Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch AIT-160-Guide-documentation-human-in-the-loop

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

mschristensen

Thanks Greg! It's a good start - overall I felt the code examples ended up being quite verbose, which makes it hard to follow in places. I thought I would put together a minimal example that includes code that I would be happy with. I've added it in a gist here: https://gist.github.com/mschristensen/c735e1f906fcad889e91190e52938273

It's not a million miles away from what you've done but I hope it's a bit more concise, and addresses some of the feedback in my comments. Please have a look at let me know if you think we can align these docs with this example.

Side point - I think for writing these guides it might make sense for us to write the code first, agree that, and then the actual docs content. Perhaps we need a repo for this? We could even make them public and link to them from guides for people to get started with that pattern. Curious to hear your thoughts on this.

mschristensen · 2026-01-22T14:11:10Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+
+## Step 1: Define a tool requiring approval <a id="step-1"/>
+
+Define an OpenAI tool that represents a sensitive operation requiring human approval. This example uses a file deletion tool that should not execute without explicit user consent.


I think we should pick a different type of tool call. Generally, we are targeting developers building cloud-hosted agents, rather than agents which run on your local machine like claude code (the requirement for client/agent sync doesn't exist in these cases, since these are no separate components that need to communicate).

Therefore I would suggest using a tool call with a more natural correspondence with operations that might be performed by a cloud agent. Perhaps something a bit more general that has a clear mapping to RBAC, e.g. publish_blog_post or similar?

mschristensen · 2026-01-22T14:15:09Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+<Code>
+```javascript
+import OpenAI from 'openai';
+import Ably from 'ably';


I think this should only be specified in the step 2 code block, where it is used (similar to other guides).

(I know there's a bit of a pain point in communicating changes to files progressively in documentation like this, I have a draft implementation of a Code component with line highlighting which I think will help, which I aim to finish up soon).

mschristensen · 2026-01-22T14:16:53Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+const channel = realtime.channels.get('ai:{{RANDOM_CHANNEL_NAME}}');
+
+// Track pending approval requests
+const pendingApprovals = new Map();


I think this should also be in the next code block in step 3, where it is used

mschristensen · 2026-01-22T14:19:38Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+  const parameters = JSON.parse(toolCall.arguments);
+  await channel.publish('approval-request', {
+    requestId,
+    tool: toolCall.name,


I the feature docs for HITL, we call this action. I actually think tool is a better name, because I think action risks confusion with the message.action, so can we please align the feature docs to use this field name too?

mschristensen · 2026-01-22T14:22:54Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+    return;
+  }
+
+  // Verify the approver (in production, check clientId or user claims)


Can we update the guide to show a concrete example of this? For example, we could use role based access, and assign an admin role to the user, and specify that only admins can delete data. I think it's quite an important part of the flow so worth describing a concrete example. Then, when we do the verification, we can show a code path which also rejects the promise.

mschristensen · 2026-01-22T14:25:29Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+
+  // Process each file with progress updates
+  const results = [];
+  for (let i = 0; i < files.length; i++) {


When we update this example to use e.g. a more generic delete_data, we can simplify this code. Instead of a loop etc, it will be clearer to just call a function like deleteData(parameters); the important bit is the surrounding code for handling the comms

mschristensen · 2026-01-22T14:30:31Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+    results.push({ file, deleted: true });
+
+    // Stream progress for each file
+    await channel.publish('tool-progress', {


We didn't cover this pattern in the HITL feature docs and I don't think we should introduce it here. However, I think it might be valid to cover this in the "Tool calls" feature docs, e.g. updates that don't come directly from the model stream, but rather from the execution of a tool by the agent. There's a variant of this that uses LiveObjects for progress % etc. So, we can remove this from here, but I have created a ticket: https://ably.atlassian.net/browse/AIT-312

mschristensen · 2026-01-22T14:37:58Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+
+<Code>
+```javascript
+async function processToolCall(toolCall) {


I don't really know how this function is adding value other than calling requestApproval and executing the tool call. I think the rest just adds cognitive overhead. Can we simplify the code as much as possible?

mschristensen · 2026-01-22T14:46:53Z

src/pages/docs/guides/ai-transport/anthropic-human-in-the-loop.mdx

+const anthropic = new Anthropic();
+
+// Define a tool that requires human approval
+const tools = [


Rather than defining static data, I think we should show the code that calls the model at this step, i.e. invoking the model the first time with the tools. I think generally we should aim to show how the pieces are used in context with the code that actually does stuff, rather than leaving random bits out in isolation. Wdyt?

mschristensen · 2026-01-22T15:40:50Z

src/pages/docs/guides/ai-transport/anthropic-human-in-the-loop.mdx

+<Code>
+```javascript
+async function requestApproval(toolUse) {
+  const requestId = crypto.randomUUID();


We dont need to generate an ID, since OpenAI gives us a tool call id

SEE: #3133 (review)

Add two new AI Transport guides demonstrating how to implement human-in-the-loop approval workflows for AI agent tool calls: - openai-human-in-the-loop.mdx: HITL with OpenAI function calling - anthropic-human-in-the-loop.mdx: HITL with Anthropic tool use Both guides cover: - Defining tools requiring human approval - Publishing approval requests with requestId correlation - Subscribing to and processing approval decisions - Progress streaming during long-running tool execution - Handling rejection gracefully

zknill · 2026-02-17T15:49:12Z

src/pages/docs/guides/ai-transport/anthropic-human-in-the-loop.mdx

+
+This guide shows you how to implement a human-in-the-loop (HITL) approval workflow for AI agent tool calls using Anthropic's Claude and Ably. The agent requests human approval before executing sensitive operations, with role-based access control to verify approvers have sufficient permissions.
+
+Using Ably for HITL workflows enables reliable, realtime communication between Claude-powered agents and human approvers. The request-response pattern ensures approval requests are delivered and decisions are processed with proper authorization checks.
+
+<Aside data-type="further-reading">
+To learn more about human-in-the-loop patterns and verification strategies, see the [human-in-the-loop](/docs/ai-transport/messaging/human-in-the-loop) documentation.
+</Aside>
+
+## Prerequisites <a id="prerequisites"/>


I think we need a section at the top here, explaining the mental model of how this guide works.

As a first time reader, I'm presented with a lot of code and changing code, without an understanding of what's happening.

I think we need to say something like:

The guide shows how to give the OpenAI model a tool that will publish a blog post. This tool implements the human approval by sending an 'approval-request' message on the channel, and waiting for an 'approval-response' with acceptable user claims before continuing to publish the blog post.

I think this is important because while we do touch on the fact that it's request-response, we don't explicitly say that it's the tool implementation that is checking the human approval.

zknill · 2026-02-17T17:44:17Z

src/pages/docs/guides/ai-transport/anthropic-human-in-the-loop.mdx

 This guide shows you how to implement a human-in-the-loop (HITL) approval workflow for AI agent tool calls using Anthropic's Claude and Ably. The agent requests human approval before executing sensitive operations, with role-based access control to verify approvers have sufficient permissions.

-Using Ably for HITL workflows enables reliable, realtime communication between Claude-powered agents and human approvers. The request-response pattern ensures approval requests are delivered and decisions are processed with proper authorization checks.
+When Claude calls a tool that requires human approval, the tool implementation itself handles the approval check before executing. Rather than executing immediately, the tool publishes an `approval-request` message to an Ably channel, waits for an `approval-response` from a human approver, verifies the approver has the required role using claims embedded in their JWT token, and only then executes the action. Claude calls the tool as normal, and the approval logic lives inside the tool's implementation.


I don't think this should be claude specific. We're talking about interacting with Anthropic models, like Opus, etc.

I think the jwt user claims needs cross link: https://ably.com/docs/ai-transport/sessions-identity/identifying-users-and-agents#user-claims

And the openai version of this paragraph is much better.

zknill · 2026-02-17T17:58:57Z

src/pages/docs/guides/ai-transport/anthropic-human-in-the-loop.mdx

+```javascript
+async function processToolUse(toolUse) {
+  if (toolUse.name === 'publish_blog_post') {
+    await requestHumanApproval(toolUse);


This line needs a comment explainer I think.

Mike's got a comment about how insufficient roles result in promise rejection (the rejection path).. we can do that here with:

// requestHumanApproval returns a promise that resolves when the human has approved // the tool use, or is rejected if the human explicitly rejects the tool call with a "decision" // other than 'approved' or the userClaims do not have the minimum role required to approve. await requestHumanApproval(toolUser)

zknill · 2026-02-17T17:59:39Z

src/pages/docs/guides/ai-transport/openai-human-in-the-loop.mdx

+When the model calls a tool that requires human approval, the tool implementation itself handles the approval check before executing. Rather than executing immediately, the tool publishes an `approval-request` message to an Ably channel, waits for an `approval-response` from a human approver, verifies the approver has the required role using claims embedded in their JWT token, and only then executes the action. The model calls the tool as normal, and the approval logic lives inside the tool's implementation.
+


cross link to jwt userclaims docs

GregHolmes self-assigned this Jan 21, 2026

GregHolmes force-pushed the AIT-160-Guide-documentation-human-in-the-loop branch 3 times, most recently from 72b74bf to de294d2 Compare January 21, 2026 11:29

GregHolmes requested a review from mschristensen January 21, 2026 11:29

GregHolmes added javascript Pull requests that update Javascript code review-app Create a Heroku review app and removed javascript Pull requests that update Javascript code labels Jan 21, 2026

ably-ci temporarily deployed to ably-docs-ait-160-guide-seflvd January 21, 2026 11:30 Inactive

mschristensen requested changes Jan 22, 2026

View reviewed changes

GregHolmes force-pushed the AIT-160-Guide-documentation-human-in-the-loop branch from de294d2 to 20c1aa3 Compare January 23, 2026 11:39

GregHolmes temporarily deployed to ably-docs-ait-160-guide-seflvd January 23, 2026 11:39 Inactive

mschristensen added a commit that referenced this pull request Jan 23, 2026

ait/hitl: align with gist developed for guide

0937fb4

SEE: #3133 (review)

mschristensen mentioned this pull request Jan 23, 2026

Bugfix/ait misc fixes #3150

Merged

3 tasks

mschristensen added a commit that referenced this pull request Jan 23, 2026

ait/hitl: align with gist developed for guide

81a5fe1

SEE: #3133 (review)

mschristensen added a commit that referenced this pull request Jan 23, 2026

ait/hitl: align with gist developed for guide

1665b75

SEE: #3133 (review)

GregHolmes force-pushed the AIT-160-Guide-documentation-human-in-the-loop branch from 20c1aa3 to 0fde30f Compare January 26, 2026 09:24

GregHolmes temporarily deployed to ably-docs-ait-160-guide-seflvd January 26, 2026 09:24 Inactive

GregHolmes temporarily deployed to ably-docs-ait-160-guide-seflvd January 26, 2026 11:10 Inactive

GregHolmes requested a review from mschristensen January 26, 2026 11:12

GregHolmes temporarily deployed to ably-docs-ait-160-guide-seflvd January 26, 2026 11:12 Inactive

matt423 mentioned this pull request Jan 30, 2026

Add progress updates documentation for tool calls (AIT-312) #3167

Merged

3 tasks

GregHolmes force-pushed the AIT-160-Guide-documentation-human-in-the-loop branch from 666d601 to 59cffdd Compare February 9, 2026 18:13

mschristensen added review-app Create a Heroku review app and removed review-app Create a Heroku review app labels Feb 9, 2026

ably-ci temporarily deployed to ably-docs-ait-160-guide-qcudzs February 9, 2026 19:28 Inactive

zknill self-requested a review February 12, 2026 16:27

GregHolmes added 2 commits February 17, 2026 14:08

fixup! docs: add human-in-the-loop guides for OpenAI and Anthropic

d0e7b4e

docs: use 'tool' field name in HITL feature docs for consistency

aece7cb

GregHolmes force-pushed the AIT-160-Guide-documentation-human-in-the-loop branch from 59cffdd to aece7cb Compare February 17, 2026 14:08

GregHolmes requested a review from kaschula February 17, 2026 15:12

zknill reviewed Feb 17, 2026

View reviewed changes

fixup! docs: add human-in-the-loop guides for OpenAI and Anthropic

de565a3

zknill reviewed Feb 17, 2026

View reviewed changes


		## Step 1: Define a tool requiring approval <a id="step-1"/>

		Define an OpenAI tool that represents a sensitive operation requiring human approval. This example uses a file deletion tool that should not execute without explicit user consent.

		When the model calls a tool that requires human approval, the tool implementation itself handles the approval check before executing. Rather than executing immediately, the tool publishes an `approval-request` message to an Ably channel, waits for an `approval-response` from a human approver, verifies the approver has the required role using claims embedded in their JWT token, and only then executes the action. The model calls the tool as normal, and the approval logic lives inside the tool's implementation.

Conversation

GregHolmes commented Jan 21, 2026

Uh oh!

coderabbitai bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

mschristensen left a comment

Choose a reason for hiding this comment

Uh oh!

mschristensen Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Comments

coderabbitai bot commented Jan 21, 2026 •

edited

Loading

mschristensen Jan 22, 2026 •

edited

Loading