You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit introduces a new developer documentation hub under the `devdocs` directory. The goal of this new
documentation is to provide a comprehensive resource for developers working on the Firestore JS SDK.
The new documentation includes:
- **Overview:** A high-level overview of the SDK, its goals, and architecture.
- **Prerequisites:** A guide for new contributors, outlining the necessary skills and knowledge.
- **Architecture:** A detailed explanation of the SDK's architecture, core components, and data flow.
- **Code Layout:** A document that explains the structure of the codebase.
- **Build & Testing:** Initial documents for the build and testing processes.
This folder contains the authoritative and official JavaScript SDK for the [Google Cloud Firestore](https://firebase.google.com/docs/firestore) database.
2
+
This project is the official JavaScript SDK for the [Google Cloud Firestore](https://firebase.google.com/docs/firestore) database.
Copy file name to clipboardExpand all lines: packages/firestore/devdocs/architecture.md
+31-51Lines changed: 31 additions & 51 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,41 +2,6 @@
2
2
3
3
This document provides a detailed explanation of the Firestore JavaScript SDK's architecture, its core components, and the flow of data through the system.
4
4
5
-
## Core Components
6
-
7
-
The SDK is composed of several key components that work together to provide the full range of Firestore features.
8
-
9
-

10
-
11
-
***API Layer**: The public-facing API surface that developers use to interact with the SDK. This layer is responsible for translating the public API calls into the internal data models and passing them to the appropriate core components.
12
-
***Core**:
13
-
***Event Manager**: Acts as a central hub for all eventing in the SDK. It is responsible for routing events between the API Layer and Sync Engine. It manages query listeners and is responsible for raising snapshot events, as well as handling connectivity changes and some query failures.
14
-
***Sync Engine**: The central controller of the SDK. It acts as the glue between the Event Manager, Local Store, and Remote Store. Its responsibilities include:
15
-
* Coordinating and translating client requests and remote events from the backend.
16
-
* Initiating responses to user code from both remote events (backend updates) and local events (e.g. garbage collection).
17
-
* Managing a "view" for each query, which represents the unified view between the local and remote data stores.
18
-
* Deciding whether a document is in a "limbo" state (e.g. its state is unknown) and needs to be fetched from the backend.
19
-
* Notifying the Remote Store when the Local Store has new mutations that need to be sent to the backend.
20
-
* For web clients, synchronizing query and mutation states across multiple tabs.
21
-
***Local Store**: A container for the components that manage persisted and in-memory data.
22
-
***Remote Table**: A cache of the most recent version of documents as known by the Firestore backend.
23
-
***Mutation Queue**: A queue of all the user-initiated writes (set, update, delete) that have not yet been acknowledged by the Firestore backend.
24
-
***Local View**: A cache that represents the user's current view of the data, combining the Remote Table with the Mutation Queue.
25
-
***Overlays**: A performance-optimizing cache that stores the calculated effect of pending mutations from the Mutation Queue on documents. Instead of re-applying mutations every time a document is read, the SDK computes this "overlay" once and caches it, allowing the Local View to be constructed more efficiently.
26
-
***Remote Store**: The component responsible for all network communication with the Firestore backend. It manages the gRPC streams for reading and writing data, and it abstracts away the complexities of the network protocol from the rest of the SDK.
27
-
***Persistence Layer**: The underlying storage mechanism used by the Local Store to persist data on the client. In the browser, this is implemented using IndexedDB.
28
-
29
-
The architecture and systems within the SDK map closely to the directory structure, which helps developers navigate the codebase. Here is a mapping of the core components to their corresponding directories.
30
-
31
-
*`src/`:
32
-
*`api/`: Implements the **API Layer** for the main SDK.
33
-
*`lite-api/`: Implements the **API Layer** for the lite SDK.
34
-
*`core/`: Implements the **Sync Engine** and **Event Manager**.
35
-
*`local/`: Implements the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, **Overlays** and the **Persistence Layer**.
36
-
*`remote/`: Implements the **Remote Store**, handling all network communication.
37
-
38
-
For a more detailed explanation of the contents of each directory, see the [Code Layout](./code-layout.md) documentation.
39
-
40
5
## Overview of features
41
6
42
7
At a high level, all interactions with Firestore can be categorized as either reading or writing data. The SDK provides different mechanisms for these operations, each with distinct guarantees and performance characteristics. There is also a special case of writing data called tansactions detailed below.
@@ -58,17 +23,39 @@ All data modifications—creates, updates, and deletes—are treated as "writes.
58
23
59
24
***One-Time Writes**: When a user performs a write (create, update, or delete), the operation is not sent directly to the backend. Instead, it's treated as a "mutation" and added to the local **Mutation Queue**. The SDK "optimistically" assumes the write will succeed on the backend and immediately reflects the change in the local view of the data, making the change visible to local queries. The SDK then works to synchronize this queue with the backend. This design is crucial for supporting offline functionality, as pending writes can be retried automatically when network connectivity is restored.
60
25
61
-
***Transactions**: For grouping multiple write operations into a single atomic unit, the SDK provides `runTransaction`. Unlike standard writes, transactions do not use the optimistic, offline-capable write pipeline. Instead, they are sent directly to the backend, which requires an active internet connection. This ensures atomicity but means transactions do not benefit from the offline capabilities of the standard write pipeline.
26
+
***Transactions**: This allows you to perform multiple writes as a single atomic operation. Transactions are immediately sent to the server, are **not** guaranteed consistnt with one-time writes, and bypass the other SDK systems.
62
27
63
-
### Data Bundles
64
28
65
-
A Firestore data bundle is a serialized collection of documents and query results, created on a server using the Firebase Admin SDK. Bundles are used to efficiently deliver a pre-packaged set of data to the client, which can then be loaded directly into the SDK's local cache. This is useful for:
29
+
## Core Components
30
+
31
+
The SDK is composed of several key components that work together to provide the full range of Firestore features.
32
+
33
+

34
+
35
+
***API Layer**: The public-facing API surface that developers use to interact with the SDK.
36
+
***Core**:
37
+
***Event Manager**: Acts as a central hub for all eventing in the SDK. It is responsible for routing events between the API Layer and Sync Engine. It manages query listeners and is responsible for raising snapshot events, as well as handling connectivity changes and some query failures.
38
+
***Sync Engine**: The central controller of the SDK. It acts as the glue between the Event Manager, Local Store, and Remote Store. Its responsibilities include:
39
+
* Coordinating client requests and remote events.
40
+
* Managing a view for each query, which represents the unified view between the local and remote data stores.
41
+
* Notifying the Remote Store when the Local Store has new mutations that need to be sent to the backend.
42
+
***Local Store**: A container for the components that manage persisted and in-memory data.
43
+
***Remote Table**: A cache of the most recent version of documents as known by the Firestore backend.
44
+
***Mutation Queue**: A queue of all the user-initiated writes (set, update, delete) that have not yet been acknowledged by the Firestore backend.
45
+
***Local View**: A cache that represents the user's current view of the data, combining the Remote Table with the Mutation Queue.
46
+
***Remote Store**: The component responsible for all network communication with the Firestore backend.
47
+
***Persistence Layer**: The underlying storage mechanism (i.e. IndexedDB) used by the Local Store.
48
+
49
+
The architecture and systems within the SDK map closely to the directory structure, which helps developers navigate the codebase. Here is a mapping of the core components to their corresponding directories.
66
50
67
-
***Optimizing Server-Side Rendering (SSR)**: Bundles enable efficient hydration of client-side applications with data pre-fetched during server-side rendering, reducing initial load times and backend roundtrips.
68
-
***Seeding initial data** for an application, allowing users to have a complete offline experience on their first use.
69
-
***Distributing curated datasets** to clients in a single, efficient package.
51
+
*`src/`:
52
+
*`api/`: Implements the **API Layer** for the main SDK.
53
+
*`lite-api/`: Implements the **API Layer** for the lite SDK.
54
+
*`core/`: Implements the **Sync Engine** and **Event Manager**.
55
+
*`local/`: Implements the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, and the **Persistence Layer**.
56
+
*`remote/`: Implements the **Remote Store**, handling all network communication.
70
57
71
-
When a bundle is loaded, its contents are unpacked and stored in the local cache, making the data available for immediate querying without needing to connect to the Firestore backend. For more details, see the [Bundles documentation](./bundles.md).
58
+
For a more detailed explanation of the contents of each directory, see the [Code Layout](./code-layout.md) documentation.
72
59
73
60
74
61
# Data Flow
@@ -80,7 +67,7 @@ Here's a step-by-step walkthrough of how data flows through the SDK for a write
80
67
1.**API Layer**: A user initiates a write operation (e.g., `setDoc`, `updateDoc`, `deleteDoc`).
81
68
2.**Sync Engine**: The call is routed to the Sync Engine, which wraps the operation in a "mutation".
82
69
3.**Mutation Queue (in Local Store)**: The Sync Engine adds this mutation to the Mutation Queue. The queue is persisted to the **Persistence Layer** (IndexedDB). At this point, the SDK "optimistically" considers the write successful locally.
83
-
4.**Local View (in Local Store)**: The change is reflected in the Local View. This is done by creating or updating a cached **Overlay** for the affected document, making the change efficiently available to any active listeners without waiting for backend confirmation.
70
+
4.**Local View (in Local Store)**: The change is immediately reflected in the Local View, making it available to any active listeners without waiting for backend confirmation.
84
71
5.**Remote Store**: The Sync Engine notifies the Remote Store that there are pending mutations.
85
72
6.**Backend**: The Remote Store sends the mutations from the queue to the Firestore backend.
86
73
7.**Acknowledgement**: The backend acknowledges the write.
@@ -91,18 +78,11 @@ Here's a step-by-step walkthrough of how data flows through the SDK for a write
91
78
1.**API Layer**: A user attaches a listener to a query (e.g., `onSnapshot`).
92
79
2.**Event Manager**: The Event Manager creates a listener and passes it to the Sync Engine.
93
80
3.**Sync Engine**: The Sync Engine creates a "view" for the query.
94
-
4.**Local View (in Local Store)**: The Sync Engine asks the Local Store for the current documents matching the query. The Local Store provides these by applying cached **Overlays** on top of the documents to reflect optimistic local changes from the **Mutation Queue**.
81
+
4.**Local View (in Local Store)**: The Sync Engine asks the Local Store for the current documents matching the query. This includes any optimistic local changes from the **Mutation Queue**.
95
82
5.**API Layer**: The initial data from the Local View is sent back to the user's `onSnapshot` callback. This provides a fast, initial result.
96
83
6.**Remote Store**: Simultaneously, the Sync Engine instructs the Remote Store to listen to the query on the Firestore backend.
97
84
7.**Backend**: The backend returns the initial matching documents for the query.
98
85
8.**Remote Table (in Local Store)**: The Remote Store receives the documents and saves them to the Remote Table in the Local Store, overwriting any previously cached versions of those documents.
99
86
9.**Sync Engine**: The Sync Engine is notified of the updated documents. It re-calculates the query view by combining the new data from the Remote Table with any applicable pending mutations from the **Mutation Queue**.
100
87
10.**API Layer**: If the query results have changed after this reconciliation, the new results are sent to the user's `onSnapshot` callback. This is why a listener may fire twice initially.
101
88
11.**Real-time Updates**: From now on, any changes on the backend that affect the query are pushed to the Remote Store, which updates the Remote Table, triggering the Sync Engine to re-calculate the view and notify the listener.
102
-
103
-
## Bundle Loading Data Flow
104
-
105
-
1.**API Layer**: The user initiates a bundle load via the public API.
106
-
2.**Sync Engine**: The Sync Engine receives the bundle and begins processing it.
107
-
3.**Local Store**: The Sync Engine unpacks the bundle and saves its contents (documents and named queries) into the **Local Store**.
108
-
4.**API Layer**: The user is notified of the progress and completion of the bundle loading operation via the task returned by the API.
Copy file name to clipboardExpand all lines: packages/firestore/devdocs/code-layout.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ This document explains the code layout in this repository. It is closely related
6
6
*`api/`: Implements the **API Layer** for the main SDK.
7
7
*`lite-api/`: Contains the entry point of for the lite SDK.
8
8
*`core/`: Contains logic for the **Sync Engine** and **Event Manager**.
9
-
*`local/`: Contains the logic the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, **Overlays**, and the **Persistence Layer**.
9
+
*`local/`: Contains the logic the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, and the **Persistence Layer**.
10
10
*`remote/`: Contains the logic for the **Remote Store**, handling all network communication.
11
11
*`model/`: Defines the internal data models used throughout the SDK, such as `Document`, `DocumentKey`, and `Mutation`. These models are used to represent Firestore data and operations in a structured way.
12
12
*`platform/`: Contains platform-specific code to abstract away the differences between the Node.js and browser environments. This includes things like networking, storage, and timers. This allows the core logic of the SDK to be platform-agnostic.
Copy file name to clipboardExpand all lines: packages/firestore/devdocs/overview.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Firestore JavaScript SDK Overview
2
2
3
-
This document is the starting point for navigating the Firestore JavaScript SDK codebase and its documentation. It provides a high-level overview of the SDK, its architecture, how it is built, tested, and the developer workflow.
3
+
This document is the starting point for navigating the Firestore JavaScript SDK codebase documentation. It provides a high-level overview of the SDK, how it is built, tested, and the developer workflow.
4
4
5
5
All contributors are expected to be familiar with the [prerequisites](./prerequisites.md) before working in this codebase.
0 commit comments