DOC-13422-Eventing-Memory-Quota #4055
Conversation
| @@ -0,0 +1,168 @@ | |||
| = Eventing Memory Quota | |||
|
|
|||
| :description: The Eventing Service memory quota does not enforce a hard memory limit on the Eventing subsystem, including producer and worker processes. | |||
There was a problem hiding this comment.
including producer and worker processes
End-user / customer doesn't know what is the "producer" process.
Please go through this entire docs page and remove all mentions of "producer" processes.
Let's only keep using the term "worker processes" (or simply "workers") as the customer / end user is only aware about that.
|
|
||
| == Overview | ||
|
|
||
| The Couchbase Eventing Service memory quota does not enforce a hard memory limit on the entire Eventing subsystem, including producer and worker processes. |
There was a problem hiding this comment.
including producer and worker processes
Same comment as above on the use of "producer" process.
|
|
||
| The memory quota serves 2 primary purposes: | ||
|
|
||
| * **Producer-to-worker queue sizing**: controls the maximum size of each worker's input queue. |
There was a problem hiding this comment.
Producer-to-worker queue sizing
Same comment as above on the use of "producer" process.
Just use "Worker queue sizing"
|
|
||
| The per-worker quota directly influences 2 key aspects of Eventing runtime behavior. | ||
|
|
||
| === Producer-to-Worker Queue Sizing |
There was a problem hiding this comment.
Producer-to-Worker Queue Sizing
Same comment as above on the use of "producer" process.
Just use "Worker queue sizing"
|
|
||
| === Producer-to-Worker Queue Sizing | ||
|
|
||
| Each worker has a bounded input queue that receives changes to documents from the Database Change Protocol (DCP) stream. |
There was a problem hiding this comment.
"Each worker has a bounded input queue" -> "Each worker maintains a bounded input queue"
|
|
||
| === Producer-to-Worker Queue Sizing | ||
|
|
||
| Each worker has a bounded input queue that receives changes to documents from the Database Change Protocol (DCP) stream. |
There was a problem hiding this comment.
Database Change Protocol (DCP) stream.
Let's not use the term "Database Change Protocol (DCP)". I don't see it mentioned anywhere in our official documentation.
Instead use a generic term like "Document Mutation stream".
There was a problem hiding this comment.
We do have a detailed section about DCP in our docs. So, as discussed, will provide reference links to that section, xref:server:learn:clusters-and-availability/intra-cluster-replication.adoc#database-change-protocol[DCP], from here.
| Each worker has a bounded input queue that receives changes to documents from the Database Change Protocol (DCP) stream. | ||
|
|
||
| * The per-worker quota determines the maximum size of this queue. | ||
| * When the queue exceeds this size, Eventing throttles the DCP stream to prevent unbounded memory growth. |
There was a problem hiding this comment.
Eventing throttles the DCP
Same as above for "DCP".
There was a problem hiding this comment.
Have provided a link to DCP from this section.
|
|
||
| * GC frees memory occupied by unreachable objects that are no longer in use. | ||
| * GC reclaims memory only from objects that are no longer in use. | ||
| Memory held by live objects in the JSE heap remains allocated, so total usage may exceed the per-worker quota. |
There was a problem hiding this comment.
"JSE" -> "Javascript runtime" * all occurrences
There was a problem hiding this comment.
Changed it to JavaScript runtime
Pallavi-Janardhan
requested review from
RichardSmedley,
abhijpes,
osfameron and
sarahlwelton
sarahlwelton
left a comment
sarahlwelton
left a comment
There was a problem hiding this comment.
Hey @Pallavi-Janardhan , can you please confirm that you're making the changes you want to be making in this PR? I can do a more thorough review once you're sure you're committing the right changes.
Thanks!
There was a problem hiding this comment.
Why are we committing an empty file? What is the point of this? I think you've made a mistake on the folder structure in this PR, because this file should be in a pages folder if it's meant to be documentation.
There was a problem hiding this comment.
Yes, guess it was by mistake! Have removed the empty file and fixed the issue.
Pallavi-Janardhan
left a comment
Pallavi-Janardhan
left a comment
There was a problem hiding this comment.
Have placed the newly created "eventing-memory-quota.adoc" doc under the "modules/learn/pages/buckets-memory-and-storage" directory and updated the cross-ref from sizing guidelines doc to this. Also have updated the nav.adoc (navigation doc).
Kindly review and let me know your comments or approve.
|
@sarahlwelton and @simon-dew can you please have a look at this and let me know if it is ok? If yes, kindly approve so that I can merge it. Thanks |
sarahlwelton
left a comment
sarahlwelton
left a comment
There was a problem hiding this comment.
This needs a lot of work. Did you go through writing a doc plan for this work? I'm not sure that you're on the right track with how you chose to approach this, and we might have been able to guide you with a doc plan ahead of time.
| {description} | ||
|
|
||
| When you plan to deploy a Couchbase Server cluster, the most common and important question that comes up is: how many nodes do I need and what size do they need to be? | ||
| The most common and important question that comes up when you plan to deploy a Couchbase Server cluster is: how many nodes do I need and what size do they need to be? |
There was a problem hiding this comment.
| The most common and important question that comes up when you plan to deploy a Couchbase Server cluster is: how many nodes do I need and what size do they need to be? | |
| The most common and important questions you need to ask when deploying a new Couchbase Server cluster are how many nodes you need, and what size they need to be. |
| Couchbase Services are what allow you to use and maintain your data. | ||
| Deploy, maintain, and provision these services independently of each other. | ||
| This independent service model allows you to take advantage of `Multi-Dimensional Scaling`, whereby a cluster can be fine-tuned for optimal handling of emergent workload-requirements, on a service-by-service basis. |
There was a problem hiding this comment.
| Couchbase Services are what allow you to use and maintain your data. | |
| Deploy, maintain, and provision these services independently of each other. | |
| This independent service model allows you to take advantage of `Multi-Dimensional Scaling`, whereby a cluster can be fine-tuned for optimal handling of emergent workload-requirements, on a service-by-service basis. | |
| Couchbase Services allow you to access and maintain your data. | |
| You can deploy, maintain, and provision these services independently of each other. | |
| This independent service model allows you to take advantage of Multi-Dimensional Scaling. | |
| Multi-Dimensional scaling lets you fine-tune your cluster for optimal handling of changing workload requirements for each individual Couchbase Service. |
| Every service has different demands on hardware resources. | ||
| Multi-Dimensional Scaling plays an important role when sizing your Couchbase cluster, both pre and post-deployment. | ||
| For example, core Data Service operations can often benefit greatly from the `scale out` of smaller commodity nodes, whereas low latency operations with the Query Service may see a greater benefit from the `scale up` of hardware resources on a given node. |
There was a problem hiding this comment.
| Every service has different demands on hardware resources. | |
| Multi-Dimensional Scaling plays an important role when sizing your Couchbase cluster, both pre and post-deployment. | |
| For example, core Data Service operations can often benefit greatly from the `scale out` of smaller commodity nodes, whereas low latency operations with the Query Service may see a greater benefit from the `scale up` of hardware resources on a given node. | |
| Every Service has different demands on hardware resources. | |
| Multi-Dimensional Scaling plays an important role when sizing your Couchbase cluster, both pre and post-deployment. | |
| For example, core Data Service operations can often benefit from scaling out smaller commodity nodes. | |
| Low latency operations with the Query Service might see a greater benefit from scaling up hardware resources on a given node. |
|
|
||
| CPU:: | ||
| CPU refers to the number of cores and clock speed that are required to run your workload. | ||
| CPU specifies the number of cores and clock speed required to run your workload. |
There was a problem hiding this comment.
| CPU specifies the number of cores and clock speed required to run your workload. | |
| CPU controls the number of cores and the clock speed required to run your workload. |
| RAM is often the most crucial areas to size accurately. | ||
| Cached documents provide low-latency reads and consistently high throughput. |
There was a problem hiding this comment.
| RAM is often the most crucial areas to size accurately. | |
| Cached documents provide low-latency reads and consistently high throughput. | |
| RAM is often the most crucial area to size. | |
| Cached documents provide low-latency reads and consistently high throughput. |
|
|
||
| As a result, setting a memory quota does not guarantee that Eventing functions remain within that limit. | ||
|
|
||
| === No OS-Level Enforcement Outside cgroups |
There was a problem hiding this comment.
| === No OS-Level Enforcement Outside cgroups | |
| [#no-enforce-c-groups] | |
| === No OS-Level Enforcement Outside cgroups |
|
|
||
| === No OS-Level Enforcement Outside cgroups | ||
|
|
||
| In non-containerized environments, the operating system does not enforce the memory quota. |
There was a problem hiding this comment.
The operating system? Or Couchbase Server/the Eventing Service?
| * Users must monitor Eventing memory usage using system tools. | ||
| * Configuring an excessive number of workers can lead to memory exhaustion. | ||
| * High-memory-consuming functions can affect overall system stability. | ||
| * Eventing does not automatically stop or throttle execution when total memory exceeds the quota. |
There was a problem hiding this comment.
| * Users must monitor Eventing memory usage using system tools. | |
| * Configuring an excessive number of workers can lead to memory exhaustion. | |
| * High-memory-consuming functions can affect overall system stability. | |
| * Eventing does not automatically stop or throttle execution when total memory exceeds the quota. | |
| You must monitor the Eventing Service's memory usage using system tools. | |
| If you configure too many workers for your Eventing Functions, you can cause memory exhaustion in your cluster. | |
| High memory-consuming Eventing Functions can affect your overall system stability. | |
| Again, the Eventing Service does not automatically stop or throttle execution when total memory exceeds the quota. |
| === Memory Isolation | ||
|
|
||
| The memory quota does not isolate memory usage between Eventing functions. | ||
|
|
||
| * All workers draw from the same total memory quota. | ||
| * A memory-intensive function can reduce the memory available to other functions. | ||
| * Eventing does not support per-function memory limits or reservations. | ||
|
|
||
| == Best Practices | ||
| Follow these recommendations to effectively manage Eventing memory usage effectively and avoid operational issues. | ||
|
|
||
| === Monitor Memory Usage | ||
|
|
||
| * Use system-level monitoring tools to track actual memory consumption. | ||
| * Monitor memory usage for the Eventing process and individual workers. | ||
| * Configure alerts for memory thresholds well below system limits. | ||
|
|
||
| === Configure Worker Count Appropriately | ||
|
|
||
| * Consider the per-worker quota when determining the number of workers. | ||
| * Avoid over-provisioning workers when Eventing functions require substantial memory. | ||
| * Balance the worker count against available system memory. |
There was a problem hiding this comment.
| === Memory Isolation | |
| The memory quota does not isolate memory usage between Eventing functions. | |
| * All workers draw from the same total memory quota. | |
| * A memory-intensive function can reduce the memory available to other functions. | |
| * Eventing does not support per-function memory limits or reservations. | |
| == Best Practices | |
| Follow these recommendations to effectively manage Eventing memory usage effectively and avoid operational issues. | |
| === Monitor Memory Usage | |
| * Use system-level monitoring tools to track actual memory consumption. | |
| * Monitor memory usage for the Eventing process and individual workers. | |
| * Configure alerts for memory thresholds well below system limits. | |
| === Configure Worker Count Appropriately | |
| * Consider the per-worker quota when determining the number of workers. | |
| * Avoid over-provisioning workers when Eventing functions require substantial memory. | |
| * Balance the worker count against available system memory. | |
| [#worker-sizing-memory] | |
| === Worker Sizing Impacts Memory | |
| When configuring the memory quota for the Eventing Service, consider the per-worker quota when determining the number of workers. | |
| Avoid over-provisioning workers when Eventing Functions require substantial memory. | |
| For best results, try to balance your worker count against your available memory. | |
| [#quota-per-worker] | |
| === Quota Applies Per Worker | |
| The memory quota does not isolate memory usage between Eventing Functions. | |
| All workers draw from the same total memory quota. | |
| A memory-intensive function can reduce the memory available to other Eventing Functions. | |
| Eventing does not support per-function memory limits or reservations. | |
| == Best Practices | |
| Follow these recommendations to effectively manage Eventing memory usage: | |
| * <<monitor,>> | |
| * <<test,>> | |
| * <<use-cgroups,>> | |
| * <<size,>> | |
| * <<review,>> | |
| [#monitor] | |
| === Monitor Memory Usage | |
| * Use system-level monitoring tools to track actual memory consumption. | |
| * Monitor memory usage for the Eventing process and individual workers. | |
| * Configure alerts for memory thresholds well below system limits. |
| === Test in Representative Environments | ||
|
|
||
| * Test Eventing functions under production-like load conditions. | ||
| * Measure actual memory consumption during peak processing. | ||
| * Validate that memory usage remains within acceptable limits. | ||
|
|
||
| === Use cgroups for Enforcement | ||
|
|
||
| * Consider containerized deployments or cgroup-based memory limits in production environments. | ||
| * Container platforms such as Docker and Kubernetes provide hard memory limits. | ||
| * cgroups enforce OS-level memory restrictions. | ||
| * This approach adds a safety layer beyond the Eventing Service memory quota. | ||
|
|
||
| === Size the Memory Quota Appropriately | ||
|
|
||
| * Set the Eventing memory quota based on expected worker count and function complexity. | ||
| * Allow additional headroom on top of the 256 MB default for production workloads. | ||
| * Account for peak processing scenarios, not just average load. | ||
|
|
||
| === Review Function Memory Efficiency | ||
|
|
||
| * Optimize JavaScript code to minimize memory allocation. | ||
| * Avoid accumulating large data structures in function scope. | ||
| * Release object references when data is no longer needed. | ||
| * Profile functions to identify memory-intensive operations. |
There was a problem hiding this comment.
| === Test in Representative Environments | |
| * Test Eventing functions under production-like load conditions. | |
| * Measure actual memory consumption during peak processing. | |
| * Validate that memory usage remains within acceptable limits. | |
| === Use cgroups for Enforcement | |
| * Consider containerized deployments or cgroup-based memory limits in production environments. | |
| * Container platforms such as Docker and Kubernetes provide hard memory limits. | |
| * cgroups enforce OS-level memory restrictions. | |
| * This approach adds a safety layer beyond the Eventing Service memory quota. | |
| === Size the Memory Quota Appropriately | |
| * Set the Eventing memory quota based on expected worker count and function complexity. | |
| * Allow additional headroom on top of the 256 MB default for production workloads. | |
| * Account for peak processing scenarios, not just average load. | |
| === Review Function Memory Efficiency | |
| * Optimize JavaScript code to minimize memory allocation. | |
| * Avoid accumulating large data structures in function scope. | |
| * Release object references when data is no longer needed. | |
| * Profile functions to identify memory-intensive operations. | |
| [#test] | |
| === Test in Representative Environments | |
| * Test Eventing Functions under production-like load conditions. | |
| * Measure actual memory consumption during peak processing. | |
| * Validate that memory usage remains within acceptable limits. | |
| [#use-cgroups] | |
| === Use cgroups for Enforcement | |
| * Consider containerized deployments or cgroup-based memory limits in production environments. | |
| * Container platforms such as Docker and Kubernetes provide hard memory limits. | |
| * cgroups enforce OS-level memory restrictions. | |
| This approach adds a safety layer beyond the Eventing Service memory quota. | |
| [#size] | |
| === Size the Memory Quota Appropriately | |
| * Set the Eventing Service memory quota based on expected worker count and Eventing Function complexity. | |
| * Allow additional headroom on top of the 256{nbsp}MB default for production workloads. | |
| * Account for peak processing scenarios, not just average load. | |
| [#review] | |
| === Review Function Memory Efficiency | |
| * Optimize JavaScript code to minimize memory allocation. | |
| * Avoid accumulating large data structures in the function scope. | |
| * Release object references when data is no longer needed. | |
| * Profile Eventing Functions to identify memory-intensive operations. |
sarahlwelton
left a comment
sarahlwelton
left a comment
There was a problem hiding this comment.
This needs a lot of work. Did you go through writing a doc plan for this work? I'm not sure that you're on the right track with how you chose to approach this, and we might have been able to guide you with a doc plan ahead of time.
|
Hi @Pallavi-Janardhan Following up on the prior discussion, after further review I believe the following location is appropriate for a dedicated Eventing Memory Quota documentation page: 
|
There was a problem hiding this comment.
As discussed with @Pallavi-Janardhan today, my comments are:
- Remove
eventing-memory-quota.adocfrom this PR and revert the changes to the navigation. In the Sizing Guidelines document, keep a link toxref:eventing:eventing-memory-quota.adoc(see below) and any style changes, following Sarah's comments. - Create a new PR in the docs-devex repo to add
eventing-memory-quota.adocto theeventingmodule, as suggested by @abhijpes.
| === RAM | ||
|
|
||
| In general, the Eventing memory quota of 256 MB is sufficient for almost all workloads. | ||
| The xref:learn:buckets-memory-and-storage/eventing-memory-quota.adoc[Eventing Memory Quota] doc discusses the Eventing Service memory quota in detail. |
There was a problem hiding this comment.
| The xref:learn:buckets-memory-and-storage/eventing-memory-quota.adoc[Eventing Memory Quota] doc discusses the Eventing Service memory quota in detail. | |
| For more information about the Eventing Service memory quota, see xref:eventing:eventing-memory-quota.adoc[]. |
| Refer to this document to learn more about: | ||
|
|
||
| -- | ||
| * Eventing Service memory quota | ||
| * How Eventing Service memory quota works | ||
| * The distribution of the total memory quota across workers | ||
| * The limitations and considerations | ||
| * Best practices and recommendations | ||
| -- |
There was a problem hiding this comment.
Not sure if you need this bullet list? It's just a summary ofeventing-memory-quota.adoc. I would delete this.
DOC-13422
Created and added a new document for Eventing Memory Quota and linked it from the Sizing Guidelines document. Also modified the Sizing Guidelines doc.
Docs preview for Sizing Guidelines
and
Docs preview for Eventing Memory Quota
Preview credentials