Implement stale while revalidate (SWR) cache strategy

[drborges]

Summary

Add support to Stale-While-Revalidate (SWR) prompt cache strategy based off the Design Document: docs/future-enhancements/STALE_WHILE_REVALIDATE_DESIGN.md

The GIF below shows the new feature in action against a self-hosted production instance of Langfuse using the following config:

Langfuse.configure do |config|
  config.public_key = ENV["LANGFUSE_PUB_KEY"]
  config.secret_key = ENV["LANGFUSE_SECRET_KEY"]
  config.base_url = ENV["LANGFUSE_BASE_URL"]

  config.cache_backend = :rails
  config.cache_ttl = 2 # cache entries are fresh for 2 seconds, after that, they become stale

  # Enable SWR
  config.cache_stale_while_revalidate = true
  config.cache_stale_ttl = 2 # cache stale entries are served for 2 seconds until the expire completely (grace period).
  config.cache_refresh_threads = 5
end

📊 Impact

• Performance: Enables instant responses for 99% of requests by serving stale data while refreshing in background
• Backward Compatibility: Fully backward compatible - SWR is opt-in via configuration (requires :rails backend)
• Dependencies: Requires concurrent-ruby gem for thread pool management

Usage

Configure the client:

Langfuse.configure do |config|
  config.public_key = ENV['LANGFUSE_PUBLIC_KEY']
  config.secret_key = ENV['LANGFUSE_SECRET_KEY']
  
  config.cache_backend = :rails
  config.cache_ttl = 300 # Fresh for 5 minutes
  
  # Enable SWR
  config.cache_stale_while_revalidate = true
  config.cache_stale_ttl = 300 # Grace period: 5 more minutes
  config.cache_refresh_threads = 10 # Background thread pool size
end

Once configured, SWR works transparently:

client = Langfuse.client

# First request - populates cache
prompt = client.get_prompt("greeting") # ~100ms (API call)

# Subsequent requests while fresh
prompt = client.get_prompt("greeting") # ~1ms (cache hit)

# After cache_ttl expires but within grace period
prompt = client.get_prompt("greeting") # ~1ms (stale data + background refresh)

# Background refresh completes, next request gets fresh data
prompt = client.get_prompt("greeting") # ~1ms (fresh cache)

More details on the proposed documentation.

[drborges]

@NoahFisher I still have to deal with some rubocop offenses and so one final review of the test coverage but I could use some feedback on this draft, I also have one open question to double check in the PR description.

[drborges]

@NoahFisher, @kxzk I think this is in a good spot for some review. Let me know if there are any concerns or change suggestions you'd like me to address. Thanks.

[drborges]

My last commit is disabling Rubocop's Metrics/ClassLength for the ApiClient class but it may be interesting to perhaps move parts of the class into mixin modules to keep the file under the desired length. Thoughts @NoahFisher?

[kxzk]

@NoahFisher, @kxzk I think this is in a good spot for some review. Let me know if there are any concerns or change suggestions you'd like me to address. Thanks.

Will try and review today.

[kxzk]

Few comments but overall looks solid

[drborges]

@kxzk thanks for all the feedback. I'll address them next.

[drborges]

@kxzk I think I was able to address all your feedback. This is ready for another round of reviews. Thanks again.

[drborges]

@kxzk, @NoahFisher let me know if this look good and what the next steps would be for us getting this merged to main.

[NoahFisher]

Additional Considerations:

1. Memory Leak Potential with In-Memory Locks

In PromptCache, locks are stored in a hash but only deleted on release_lock.

If a refresh thread crashes between acquire_lock and release_lock (despite the ensure block), the lock is never released. The ensure should handle this, but if the thread is killed externally (e.g., Thread#kill), the lock persists forever. Unlike Redis locks which have TTL expiration, these in-memory locks have no timeout.

Suggestion: Accept this as a known limitation of the in-memory backend, but document it.

2. No Backpressure Signal to Callers

When the thread pool queue fills up and tasks are discarded, callers have no idea. This is fine for SWR (they get stale data), but it means:

• Monitoring is blind to refresh failures
• If the API is down for extended periods, stale data keeps being served with no signal

Question: Should there be a hook or callback for observability? e.g., on_refresh_dropped, on_refresh_failed?

3. Cold Start Behavior

On application boot with an empty cache, the first request for each prompt will be a synchronous API call (cache miss). With SWR enabled, you've documented cache warming, which is good. But worth calling out:

SWR doesn't help cold starts - it only helps after the cache has been populated at least once.

4. Ruby's GVL (Global VM Lock)

Worth noting that Ruby's GVL means these threads aren't truly parallel for CPU work. However, for I/O-bound work (HTTP calls to Langfuse API), threads do release the GVL during I/O waits, so this is fine. Just worth being aware that max_threads: 5 doesn't mean 5x throughput for CPU-bound work.

5. Should SWR be Default-On for Rails Backend?

The PR makes SWR opt-in (cache_stale_while_revalidate = false by default). Given the latency benefits and that the distributed lock prevents thundering herd, is there an argument for making
it default-on for the :rails backend specifically? Users with Redis already have the
infrastructure for distributed locks.

Counter-argument: Surprising behavior changes are bad. Opt-in is safer.

Good work so far, I wouldn't consider the above blocking comments (I feel more strongly about the inline ones), but curious if we should document our thinking for next steps.

[NoahFisher]

On Configuration Complexity

Concern: This PR adds 3 new configuration options (cache_stale_while_revalidate, cache_stale_ttl, cache_refresh_threads). I'd push back on exposing all of these to users.

Principle: Every configuration option is a decision the user has to make. Good defaults mean fewer decisions.

Suggestion: Consider collapsing to a single option:

Instead of:

config.cache_stale_while_revalidate = true
config.cache_stale_ttl = 300
config.cache_refresh_threads = 5

Just:

config.cache_stale_while_revalidate = true  # Uses sensible defaults internally

Reasoning:

• cache_stale_ttl defaulting to cache_ttl is already the right choice 99% of the time - why expose it?
• cache_refresh_threads at 5 is fine for virtually all deployments - this is an implementation detail, not a user concern
• Users who truly need to tune these can always subclass or we can add them later

The bar for adding config options should be: "Will a significant number of users need to change this from the default?" If not, hardcode it.

[benlangfeld]

We absolutely need to set the TTL to infinite. Not exposing the TTL setting won't work for us.

[NoahFisher]

Can you explain what the use case for a non-infinite TTL?

[drborges]

Every configuration option is a decision the user has to make. Good defaults mean fewer decisions.

I agree, it certainly makes sense to hide cache_refresh_threads. Exposing cache_stale_ttl would be more of a flexibility in my opinion in case clients have very specific needs.

With that said, if I choose to enable SWR, I would more often than not want an infinite grace period, e.g. cache_stale_ttl = :indefinite for high availability reasons, then use cache_ttl to control how often cache is refreshed.

Based on that, only exposing config.cache_stale_while_revalidate would make sense to me if that defaults cache_stale_ttl to :indefinite as per the proposed implementation.

Does that make sense?

[drborges]

Thanks for all of the great feedback so far @NoahFisher! I've addressed some of the points you raised and wanted to follow up on a few of the points you brought up above:

Point 2

No Backpressure Signal to Callers
When the thread pool queue fills up and tasks are discarded, callers have no idea. This is fine for SWR (they get stale data), but it means:

Monitoring is blind to refresh failures
If the API is down for extended periods, stale data keeps being served with no signal
Question: Should there be a hook or callback for observability? e.g., on_refresh_dropped, on_refresh_failed?

This is a great point and I think it would be interesting to provide such hooks. Is the following somewhat aligned with what you had in mind?

Langfuse.configure do |config|
  # Hook called whenever a refresh attempt fails for whatever reason
  config.on_cache_refresh_failed do |error, cache_key|
    # handle refresh error accordingly... e.g. publish to Sentry, log something out, etc...
  end
  
  # Hook called whenever a refresh attempt is dropped due to not being able to acquire the lock
  config.on_cache_refresh_dropped do |cache_key|
    # handle refresh drops accordingly...
  end
end

Can you think of any other scenario we'd like to cover?

Also, would it make sense to ship that as another enhancement PR?

Point 3

With SWR enabled, you've documented cache warming, which is good.

Could you point me to that doc entry so I can double check? Cache warming should not be impacted by SWR as you pointed out, so that would likely be a mishap on my part.

Point 5

Should SWR be Default-On for Rails Backend?
The PR makes SWR opt-in (cache_stale_while_revalidate = false by default). Given the latency benefits and that the distributed lock prevents thundering herd, is there an argument for making
it default-on for the :rails backend specifically? Users with Redis already have the infrastructure for distributed locks.

Counter-argument: Surprising behavior changes are bad. Opt-in is safer.

My initial thinking was exactly that "Opt-in seemed safer" and intentional. Now, you do raise a good point, if 99% of the time Rails clients end up turning SWR on, then perhaps it should be the default? I could go either way on this one to be honest.

[drborges]

@NoahFisher, @kxzk do you think we may be able to cut a new release including the work in this PR at some point this week?

Also, one small snag I just ran into is that the project we plan on using langfuse-rb already has the faraday gem pinned to 1.0 due to some old dependency limitations we yet have to deal with.

Now the simplest solution to our problem seems to be having langfuse-rb depend on faraday >= 1.0 as it does not appear to require any specific feature from faraday 2.0. Would this be something we could consider for this PR or a follow-up?

[drborges]

Moving the faraday convo to a new issue: #38

[drborges]

@NoahFisher, @kxzk any thoughts on the last update? Do you think we could move on with the solution as is and address anything that is non-blocking in a follow-up PR perhaps?

[benlangfeld]

@kxzk @NoahFisher Happy new year. Could we please get this merged? It's been going quite a long time.