docs: Adding details from life of a query training video

Author: rafikhan

packages/firestore/devdocs/architecture.md

@@ -12,13 +12,17 @@ The SDK is composed of several key components that work together to provide the
 *   **Core**:
     *   **Event Manager**: Acts as a central hub for all eventing in the SDK. It is responsible for routing events between the API Layer and Sync Engine. It manages query listeners and is responsible for raising snapshot events, as well as handling connectivity changes and some query failures.
     *   **Sync Engine**: The central controller of the SDK. It acts as the glue between the Event Manager, Local Store, and Remote Store. Its responsibilities include:
-        *   Coordinating client requests and remote events.
-        *   Managing a view for each query, which represents the unified view between the local and remote data stores.
+        *   Coordinating and translating client requests and remote events from the backend.
+        *   Initiating responses to user code from both remote events (backend updates) and local events (e.g. garbage collection).
+        *   Managing a "view" for each query, which represents the unified view between the local and remote data stores. The Sync Engine builds the user-facing "View" using the formula: `View = Remote Document + Overlay`. A **Remote Document** is the authoritative state from the backend. An **Overlay** is A computed "delta" representing pending local mutations. Overlays are calculated immediately when a mutation is applied and persisted separately. This allows for zero-latency "Optimistic Updates."
+        *   Deciding whether a document is in a "limbo" state (e.g. its state is unknown) and needs to be fetched from the backend.
         *   Notifying the Remote Store when the Local Store has new mutations that need to be sent to the backend.
 *   **Local Store**: A container for the components that manage persisted and in-memory data.
-    *   **Remote Table**: A cache of the most recent version of documents as known by the Firestore backend.
+    *   **Remote Table**: A cache of the most recent version of documents as known by the Firestore backend (A.K.A. Remote Documents).
     *   **Mutation Queue**: A queue of all the user-initiated writes (set, update, delete) that have not yet been acknowledged by the Firestore backend.
     *   **Local View**: A cache that represents the user's current view of the data, combining the Remote Table with the Mutation Queue.
+    *   **Query Engine**: Determines the most efficient strategy (Index vs. Scan) to identify documents matching a query in the local cache.
+    *   **Overlays**: A performance-optimizing cache that stores the calculated effect of pending mutations from the Mutation Queue on documents. Instead of re-applying mutations every time a document is read, the SDK computes this "overlay" once and caches it, allowing the Local View to be constructed more efficiently.
 *   **Remote Store**: The component responsible for all network communication with the Firestore backend. It manages the gRPC streams for reading and writing data, and it abstracts away the complexities of the network protocol from the rest of the SDK.
 *   **Persistence Layer**: The underlying storage mechanism used by the Local Store to persist data on the client. In the browser, this is implemented using IndexedDB.
 
@@ -77,7 +81,7 @@ Here's a step-by-step walkthrough of how data flows through the SDK for a write
 1.  **API Layer**: A user attaches a listener to a query (e.g., `onSnapshot`).
 2.  **Event Manager**: The Event Manager creates a listener and passes it to the Sync Engine.
 3.  **Sync Engine**: The Sync Engine creates a "view" for the query.
-4.  **Local View (in Local Store)**: The Sync Engine asks the Local Store for the current documents matching the query. This includes any optimistic local changes from the **Mutation Queue**.
+4.  **Local View (in Local Store)**: The Sync Engine asks the Query Engine in the Local Store to execute the query. The Query Engine selects a strategy (e.g., Index Scan or Timestamp Optimization) to find matching keys. The Local Store then constructs the documents by applying cached Overlays on top of Remote Documents.
 5.  **API Layer**: The initial data from the Local View is sent back to the user's `onSnapshot` callback. This provides a fast, initial result.
 6.  **Remote Store**: Simultaneously, the Sync Engine instructs the Remote Store to listen to the query on the Firestore backend.
 7.  **Backend**: The backend returns the initial matching documents for the query.

packages/firestore/devdocs/code-layout.md

@@ -6,7 +6,12 @@ This document explains the code layout in this repository. It is closely related
     *   `api/`: Implements the **API Layer** for the main SDK.
     *   `lite-api/`: Contains the entry point of for the lite SDK.
     *   `core/`: Contains logic for the **Sync Engine** and **Event Manager**.
-    *   `local/`: Contains the logic the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, and the **Persistence Layer**.
+    *   `local/`: Contains the logic the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, **Overlays**, and the **Persistence Layer**
+        *   `local_store.ts`: The main entry point for persistence operations.
+        *   `query_engine.ts`: Implements the strategy selection logic (Scan vs. Index).
+        *   `index_backfiller.ts`: The background task that updates Client-Side Indexes.
+        *   `remote_document_cache.ts`: Manages the `remote_documents` table (base truth).
+        *   `overlay_cache.ts`: Manages pending mutation queue.
     *   `remote/`: Contains the logic for the **Remote Store**, handling all network communication.
     *   `model/`: Defines the internal data models used throughout the SDK, such as `Document`, `DocumentKey`, and `Mutation`. These models are used to represent Firestore data and operations in a structured way.
     *   `platform/`: Contains platform-specific code to abstract away the differences between the Node.js and browser environments. This includes things like networking, storage, and timers. This allows the core logic of the SDK to be platform-agnostic.

packages/firestore/devdocs/overview.md

@@ -16,6 +16,16 @@ The primary goals of this SDK are:
 *   Offer a lightweight version for applications that do not require advanced features.
 *   Maintain API and architectural symmetry with the [Firestore Android SDK](https://github.com/firebase/firebase-android-sdk) and [Firestore iOS SDK](https://github.com/firebase/firebase-ios-sdk). This consistency simplifies maintenance and makes it easier to port features between platforms. The public API is intentionally consistent across platforms, even if it means being less idiomatic, to allow developers to more easily port their application code.
 
+
+## Key Concepts & Vocabulary
+
+*   **Query**: The client-side representation of a data request (filters, order bys).
+*   **Target**: The backend's representation of a Query. The SDK allocates a unique integer `TargetID` for every unique query to manage the real-time stream.
+*   **Mutation**: A user-initiated change (Set, Update, Delete). Mutations are queued locally and sent to the backend.
+*   **Overlay**: The computed result of applying a Mutation to a Document. We store these to show "Optimistic Updates" instantly without modifying the underlying "Remote Document" until the server confirms the write.
+*   **Limbo**: A state where a document exists locally and matches a query, but the server hasn't explicitly confirmed it belongs to the current snapshot version. The SDK must perform "Limbo Resolution" to ensure these documents are valid.
+
+
 ## Artifacts
 
 The Firestore JavaScript SDK is divided into two main packages:

packages/firestore/devdocs/query-execution.md

@@ -0,0 +1,38 @@
+# Query Execution & Indexing
+
+This document details how the Firestore SDK executes queries against the local cache. Understanding this is crucial for debugging performance issues and understanding offline behavior.
+
+## The Query Engine
+
+The **Query Engine** is the component within the **Local Store** responsible for finding the set of document keys that match a given query. It does not load the full document data; it only identifies the keys. It employs a hierarchy of strategies, ordered by efficiency:
+
+1.  **Full Index Scan (O(log N))**:
+    *   Used when a Client-Side Index (CSI) exists that covers all filters and sort orders of the query.
+    *   This is the most performant strategy.
+2.  **Partial Index Scan**:
+    *   Used when an index covers some filters (typically equality filters like `where('status', '==', 'published')`).
+    *   The engine uses the index to narrow down the potential keys and then performs a scan on that smaller subset to verify the remaining conditions.
+3.  **Index-Free (Timestamp) Optimization**:
+    *   **Concept**: If the client has been online and syncing, it knows the state of the collection up to a specific point in time (the `lastLimboFreeSnapshot`).
+    *   **Mechanism**: The SDK assumes the "base state" (documents matching at the last snapshot) is correct. It then only scans the `remote_documents` table for documents modified *after* that snapshot version.
+    *   This drastically reduces the work required for active listeners, changing the cost from *Collection Size* to *Recent Change Volume*.
+4.  **Full Collection Scan (O(N))**:
+    *   The fallback strategy. The engine iterates through every document in the collection locally to check for matches.
+
+## Client-Side Indexing (CSI)
+
+To support efficient querying without blocking the main thread, the SDK utilizes a decoupled indexing architecture.
+
+*   **Structure**: Index entries are stored in a dedicated `index_entries` table. An entry maps field values (e.g., `(coll/doc1, fieldA=1)`) to a document key.
+*   **The Index Backfiller**: Indexes are **not** updated synchronously when you write a document. Instead, a background task called the **Backfiller** runs periodically (when the SDK is idle). It reads new/modified documents and updates the index entries.
+*   **Hybrid Lookup**: Because the Backfiller is asynchronous, the index might be "stale" (behind the document cache). To guarantee consistency, the Query Engine performs a **Hybrid Lookup**:
+    1.  Query the **Index** for results up to the `IndexOffset` (the point where the Backfiller stopped).
+    2.  Query the **Remote Document Cache** for any documents modified *after* the `IndexOffset`.
+    3.  Merge the results.
+
+## Composite Queries (OR / IN)
+
+Queries using `OR` or `IN` are not executed as a single monolithic scan. The SDK transforms these into **Disjunctive Normal Form (DNF)**—essentially breaking them into multiple sub-queries.
+
+*   **Execution**: Each sub-query is executed independently using the strategies above (Index vs. Scan).
+*   **Union**: The resulting sets of document keys are unioned together in memory to produce the final result.

packages/firestore/devdocs/testing.md

@@ -9,6 +9,11 @@ This document provides a detailed explanation of the Firestore JavaScript SDK te
 - Firebase emulator for local development
 - Integration testing with the backend
 
+## Component Testing
+*   **Query Engine Tests**: We rely on specific spec tests to ensure the Query Engine picks the correct strategy (e.g., verifying that it uses an Index when available rather than scanning).
+*   **Integration Tests**: Use the Firebase Emulator to verify that "Limbo Resolution" correctly fetches documents when the local cache drifts from the server state.
+
+
 # Patterns and Practices