Standalone activity heartbeating #8730

dandavison · 2025-12-01T23:02:43Z

What changed?

Implement heartbeating for CHASM activities (standalone activity)

Why?

Required feature

How did you test it?

built
added new functional test(s)

Note

Implements heartbeat recording and timeout handling for standalone (CHASM) activities, including token validation, a heartbeat timer task, API wiring, and tests.

Activity (CHASM library):
- Add RecordHeartbeat with task token validation and response (CancelRequested), persisting last heartbeat and scheduling next heartbeatTimer.
- Introduce WithToken[T] wrapper and apply token validation to HandleCompleted/Failed/Canceled.
- Add ValidateActivityTaskToken and use in handlers.
- Schedule heartbeat timer on TransitionStarted; handle TIMEOUT_TYPE_HEARTBEAT in TransitionTimedOut.
Tasks/Executors:
- Add HeartbeatTimeoutTask proto and executor (validate deadline/attempt; reschedule or timeout; retry via TransitionRescheduled).
- Register new task in fx/library modules.
History APIs:
- Wire standalone path for RecordActivityTaskHeartbeat, RespondActivityTask{Completed,Failed,Canceled} to call CHASM with WithToken.
Protos/Generated:
- Extend proto/v1/tasks.proto and generated files for HeartbeatTimeoutTask helpers/messages.
Tests:
- Add heartbeat tests: invalid/malformed tokens, stale token/attempt, cancel-requested flag, details propagation on retry, heartbeat timeout (with/without retry), and keepalive heartbeats.
- Minor assertion message improvements in existing tests.

^{Written by Cursor Bugbot for commit 7c0893a. This will update automatically on new commits. Configure here.}

…level" This reverts commit 5412f7e. Document execution-level notifications

This reverts commit c623906.

Cleanup Add test of cross-execution token usage Fix test Change DeserializeComponentRef to return error on empty input Handle empty token same as absent wait policy Remove chasm engine from tests Cleanup Reinstate ignored opts Refactor: ErrorAs Refactor: NotFound ErrorAs Revert "Fix CHASM not found errors" This reverts commit d3c709c. Commute NotFound error in activity handler errors.New Cleanup Revert irrelevant changes Changes from code review

This reverts commit a92b69e.

This reverts commit f322ea18257a1e485d839f2af98c689c73c0f110.

dandavison · 2025-12-01T23:05:57Z

chasm/lib/activity/activity.go

+func (a *Activity) HandleStarted(
+	ctx chasm.MutableContext,
+	request *historyservice.RecordActivityTaskStartedRequest,
+) (*historyservice.RecordActivityTaskStartedResponse, error) {


ignore: formatting only

Looks like @fretz12 and you have different ideas on how to format these signatures. I personally like the style in this PR better.

I like this formatting better too, but ideally we should have the linter take care of it.
I'm running gofumpt locally, it seems to do a good job... maybe introduce it as part of the lint-code target?

dandavison · 2025-12-01T23:06:03Z

chasm/lib/activity/activity.go

+	ctx chasm.Context,
+	key chasm.EntityKey,
+	response *historyservice.RecordActivityTaskStartedResponse,
+) error {


ignore: formatting only

dandavison · 2025-12-01T23:06:42Z

chasm/lib/activity/activity.go

+func (a *Activity) handleCancellationRequested(
+	ctx chasm.MutableContext,
+	req *activitypb.CancelActivityExecutionRequest,
+) (*activitypb.CancelActivityExecutionResponse, error) {


ignore: formatting only

bergundy · 2025-12-02T00:58:26Z

chasm/lib/activity/activity.go

+func (a *Activity) HandleStarted(
+	ctx chasm.MutableContext,
+	request *historyservice.RecordActivityTaskStartedRequest,
+) (*historyservice.RecordActivityTaskStartedResponse, error) {


Looks like @fretz12 and you have different ideas on how to format these signatures. I personally like the style in this PR better.

bergundy · 2025-12-02T01:06:13Z

chasm/lib/activity/activity_tasks.go

+}
+
+// Validate validates a HeartbeatTimeoutTask.
+func (e *heartbeatTimeoutTaskExecutor) Validate(


You will want to validate that the schedule time in the attributes is still relevant here not in the execute function. It should deterministic function of the last heartbeat time and the attempt start time (hbDeadline below should be equal to the schedule time).

bergundy · 2025-12-02T01:07:11Z

chasm/lib/activity/activity_tasks.go

+	hbTimeout := activity.GetHeartbeatTimeout().AsDuration()
+	attemptStartTime := attempt.GetStartedTime().AsTime()
+	lastHbTime := lastHb.GetRecordedTime().AsTime() // could be from a previous attempt
+	// No heartbeats in the attempt so far is equivalent to a heartbeat having been sent at attempt
+	// start time.
+	hbDeadline := util.MaxTime(lastHbTime, attemptStartTime).Add(hbTimeout)
+
+	if ctx.Now(activity).Before(hbDeadline) {
+		// Deadline has not expired; schedule a new task.
+		ctx.AddTask(
+			activity,
+			chasm.TaskAttributes{
+				ScheduledTime: hbDeadline,
+			},
+			&activitypb.HeartbeatTimeoutTask{
+				Attempt: attempt.GetCount(),
+			},
+		)
+		return nil
+	}


This shouldn't happen here. Validate in the Validate function. Schedule a new task any time a heartbeat is received.

bergundy · 2025-12-02T01:10:08Z

chasm/lib/activity/library.go

 			l.activityDispatchTaskExecutor,
 			l.activityDispatchTaskExecutor,
 		),
+		// TODO(dan): why are the task names "FooTimer" but "FooTimeoutTask" in the struct names?


We agreed not to add Task to the task names. Not sure if we want to use TaskExecutor vs. just Executor for the struct names but that is not critical and can change easily. The string names cannot be changed easily OTOH because they affect how tasks are represented in persistence.

bergundy · 2025-12-02T01:14:47Z

chasm/lib/activity/activity.go

+		Details:      input.Request.HeartbeatRequest.GetDetails(),
 	})
-	return nil, nil
+	return &historyservice.RecordActivityTaskHeartbeatResponse{


Just before returning here you want to generate a new heartbeat task if the heartbeat timeout is set.

bergundy · 2025-12-02T01:17:20Z

tests/standalone_activity_test.go

+				require.Error(t, err)
+				statusErr := serviceerror.ToStatus(err)
+				require.NotNil(t, statusErr)
+				require.Equal(t, codes.InvalidArgument, statusErr.Code())


Alternatively, you can use require.ErrorAs(err, &invalidArgumentErr)

fretz12 · 2025-12-02T00:20:45Z

chasm/lib/activity/validator.go

+) error {
+	if a.Status != activitystatepb.ACTIVITY_EXECUTION_STATUS_STARTED &&
+		a.Status != activitystatepb.ACTIVITY_EXECUTION_STATUS_CANCEL_REQUESTED {
+		return serviceerror.NewNotFound("activity task not found")


Wrong error message? Also NotFound doesn't feel like the right error type, even though I get it's checking if a token "matches". Perhaps FailPrecondition?

fretz12 · 2025-12-02T00:21:29Z

chasm/lib/activity/validator.go

+		return err
+	}
+	if token.Attempt != attempt.GetCount() {
+		return serviceerror.NewNotFound("activity task not found")


Same comment as above

fretz12 · 2025-12-02T00:28:55Z

chasm/lib/activity/activity.go

+	if err := ValidateActivityTaskToken(ctx, a, input.Token); err != nil {
+		return nil, err
+	}
 	a.LastHeartbeat = chasm.NewDataField(ctx, &activitypb.ActivityHeartbeatState{


@bergundy is there any performance pentalty if we create a new field on every hearbeat?

No, no penalty.

fretz12 · 2025-12-02T00:31:22Z

chasm/lib/activity/activity.go

 	a.LastHeartbeat = chasm.NewDataField(ctx, &activitypb.ActivityHeartbeatState{
 		RecordedTime: timestamppb.New(ctx.Now(a)),
-		Details:      details,
+		Details:      input.Request.HeartbeatRequest.GetDetails(),


Suggested change

Details: input.Request.HeartbeatRequest.GetDetails(),

Details: input.Request.GetHeartbeatRequest().GetDetails(),

fretz12 · 2025-12-02T01:00:43Z

chasm/lib/activity/activity_tasks.go

+func (e *heartbeatTimeoutTaskExecutor) Execute(
+	ctx chasm.MutableContext,
+	activity *Activity,
+	taskAttrs chasm.TaskAttributes,


nit: taskAttrs, task are unused

fretz12 · 2025-12-02T01:09:03Z

chasm/lib/activity/activity_tasks.go

+		ctx.AddTask(
+			activity,
+			chasm.TaskAttributes{
+				ScheduledTime: hbDeadline,


If hbDeadline happens to be attemptStartTime + hbTimeout (i.e., no hearbeat recorded yet), won't the next timer basically pop the same time as the current one being executed?

- Hearbeats always add a task - Task is invalid if lastHeartbeat (or activity start) is within timeout - Otherwise task fails attempt

cursor · 2025-12-05T01:08:54Z

chasm/lib/activity/activity.go

+		&activitypb.HeartbeatTimeoutTask{
+			Attempt: attempt.GetCount(),
+		},
+	)


Bug: Heartbeat task scheduled even when timeout is zero

The RecordHeartbeat function unconditionally schedules a HeartbeatTimeoutTask regardless of whether a heartbeat timeout is configured. When HeartbeatTimeout is zero or not set, ctx.Now(a).Add(a.GetHeartbeatTimeout().AsDuration()) schedules a task for "now", which is incorrect behavior. The code in TransitionStarted correctly guards this with if heartbeatTimeout := a.GetHeartbeatTimeout().AsDuration(); heartbeatTimeout > 0 before adding the task. The same guard is needed here.

dandavison added 30 commits November 12, 2025 18:12

PollComponent and PollActivityExecution

f8ccafb

Changes from @fretz12's review

46eb18e

Edit docstrings

11cac65

Non-error response on long-poll timeout

11c6b59

Remove unused opts parameter of PollComponent

b24a022

Error on runID mismatch

0db9858

Revert comment

c888969

Use framework-provided predicate to wait for any state change

5740320

Address code review comments

1d27df4

Use component last updated VT

1107a99

Cleanup

2ee2b75

Refactor

ea12fcc

Refcator: combine handlers

5c2c41e

Rename: ComponentStateToken

c02c869

entity -> execution

e1a52d4

Rename: ref.componentVT

24ef87c

Subscribe to notifications at component level, not execution level

5412f7e

Revert "Subscribe to notifications at component level, not execution …

612426f

…level" This reverts commit 5412f7e. Document execution-level notifications

Add non-functional test

7958efd

Add tests of cross-component subscribe/notify

3904355

Rename: ComponentStateChanged

00ef0ff

Revert to using ref as long-poll token

da60095

Rename: ChasmExecutionNotification

6d5fc28

Functional test env

a604d80

DEBUG

6917450

TestStartToClose

27aafa7

ScheduleToStart timeout

c623906

Revert "ScheduleToStart timeout"

e05af5a

This reverts commit c623906.

Evolve test

6bd6d57

Evolve test

f325df8

dandavison added 7 commits December 1, 2025 15:30

Revert "changes from poll-component"

f180a26

This reverts commit a92b69e.

Improve test failure messages

5b7828c

Formatting

aca3952

Functional tests

bd9cb2a

CHASM activity heartbeat handler

d2f0301

Rename: getOrCreateLastHeartbeat

da4ebd9

dandavison requested review from a team as code owners December 1, 2025 23:02

dandavison requested review from bergundy and fretz12 December 1, 2025 23:03

dandavison force-pushed the heartbeat branch from 4b24ef4 to 0527002 Compare December 1, 2025 23:11

dandavison added 5 commits December 1, 2025 18:22

Implement heartbeat timer task logic

1527938

Mark test as TODO

b796b64

Revert "Mark test as TODO"

32b8798

This reverts commit f322ea18257a1e485d839f2af98c689c73c0f110.

Token validation

32270ef

Test heartbeating with stale token (non-current attempt)

6ffe4da

dandavison force-pushed the heartbeat branch from 0527002 to 6ffe4da Compare December 1, 2025 23:22

dandavison commented Dec 1, 2025

View reviewed changes

Add doc comments

fce1b3f

bergundy reviewed Dec 2, 2025

View reviewed changes

fretz12 reviewed Dec 2, 2025

View reviewed changes

dandavison added 2 commits December 4, 2025 12:25

Mark parameters as unused

979480a

Change heartbeating algorithm

8e226be

- Hearbeats always add a task - Task is invalid if lastHeartbeat (or activity start) is within timeout - Otherwise task fails attempt

cursor bot reviewed Dec 5, 2025

View reviewed changes

Cleanup

7c0893a

dandavison force-pushed the saa-schedule-to-close-bug-2 branch 3 times, most recently from 81b2516 to c8e5800 Compare December 6, 2025 00:24

Base automatically changed from saa-schedule-to-close-bug-2 to standalone-activity December 6, 2025 00:31

	Details: input.Request.HeartbeatRequest.GetDetails(),
	Details: input.Request.GetHeartbeatRequest().GetDetails(),

Standalone activity heartbeating #8730

Are you sure you want to change the base?

Standalone activity heartbeating #8730

Uh oh!

Conversation

dandavison commented Dec 1, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How did you test it?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Dec 5, 2025

Choose a reason for hiding this comment

Bug: Heartbeat task scheduled even when timeout is zero

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dandavison commented Dec 1, 2025 •

edited by cursor bot

Loading