Skip to content

Commit 1201af2

Browse files
darthunixdarthunix
committed
doc: implement a memory bank for AI agents
1 parent d764db7 commit 1201af2

20 files changed

+640
-21
lines changed

‎README.md‎

Lines changed: 67 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,46 @@
11
# pg_fusion
22

3-
Currently, PostgreSQL operates a row-based engine built on the Volcano
4-
architecture. While this is a good solution for OLTP workloads, it is
5-
very slow for analytical processing. Modern OLAP engines leverage columnar
6-
data representation in memory to enable SIMD optimizations and take advantage
7-
of data locality in CPU caches.
3+
pg_fusion is a PostgreSQL extension that delegates query execution to an external Apache DataFusion runtime. The extension hooks PG planning/execution, streams heap pages via shared memory to the runtime, and streams back results as wire‑friendly MinimalTuple frames.
84

9-
Additionally, PostgreSQL operates on a process-based model, where each process
10-
executes a single thread. The issue with this approach is that these threads
11-
handle not only data processing but also I/O tasks: reading blocks from disk
12-
into a shared memory cache, and communicating with clients over the network.
13-
As a result, the operating system scheduler frequently removes threads from CPU
14-
cores and later reinstates them, which negatively impacts TLB efficiency.
5+
Why: PostgreSQL’s Volcano (row‑at‑a‑time) engine is great for OLTP but slow for OLAP. DataFusion provides a modern, vectorized execution engine in Rust. pg_fusion integrates it in‑process via a background worker and shared memory IPC.
156

16-
These limitations make historical analytics in PostgreSQL a slow and cumbersome
17-
process. In comparison, DataFusion processes queries an order of magnitude faster.
18-
This leads to the hypothesis that PostgreSQL users need a CPU-efficient engine
19-
capable of significantly accelerating most read-heavy queries on heap tables.
20-
This is the motivation behind the development of `pg_fusion`.
7+
## Architecture (high level)
218

22-
## How to run
9+
- Extension (pgrx) sits in the backend process and drives Parse/Bind/Optimize/Translate/Begin/Exec/End.
10+
- Executor (runtime) runs DataFusion; `PgTableProvider → PgScanExec → PgScanStream` scans heap pages from shared memory and produces Arrow RecordBatches.
11+
- Protocol defines control/data packets and a compact wire tuple format with explicit alignment.
2312

24-
After installing `postgres`, you need to set up `rustup`, `cargo-pgrx` to build
25-
the extension.
13+
See: `ai/memory/architecture.md` and component notes in `ai/memory/components/`.
14+
15+
## Repository layout
16+
17+
- `postgres/` — pgrx extension (hooks, IPC, TupleTableSlot fill)
18+
- `executor/` — DataFusion runtime (planning/execution, SHM access, result encoding)
19+
- `protocol/` — shared packets and wire formats
20+
- `storage/` — heap page reader + zero‑allocation tuple decoder to DataFusion `ScalarValue`
21+
- `common/` — shared errors/types
22+
23+
## Build & test
24+
25+
Workspace targets Rust 1.89.
26+
27+
Basics:
28+
29+
```
30+
cargo check --workspace
31+
cargo test --workspace
32+
```
33+
34+
Lint/format:
35+
36+
```
37+
cargo fmt --all
38+
cargo clippy --all-targets --features "pg17, pg_test" --no-default-features
39+
```
40+
41+
## pgrx quickstart (PG 17)
42+
43+
Install and initialize pgrx:
2644

2745
```
2846
# install rustup
@@ -34,9 +52,37 @@ cargo install cargo-pgrx
3452
# configure pgrx
3553
cargo pgrx init --pg17 $(which pg_config)
3654
37-
# append the extension to shared_preload_libraries in ~/.pgrx/data-17/postgresql.conf
55+
# enable extension in the dev cluster
3856
echo "shared_preload_libraries = 'pg_fusion'" >> ~/.pgrx/data-17/postgresql.conf
3957
40-
# run cargo-pgrx to build and install the extension
58+
# run the dev cluster and build/install the extension
4159
cargo pgrx run
4260
```
61+
62+
Extension‑specific commands:
63+
64+
```
65+
# build only the extension crate
66+
cargo build -p pg_fusion
67+
68+
# run pgrx tests for storage’s pg_test
69+
cargo pgrx test pg17 -p pg_test
70+
```
71+
72+
## Developer guidelines
73+
74+
- Rust 2021; keep changes small and focused; surface structured errors (no panics in extension paths).
75+
- Before PR: `cargo fmt`, `cargo clippy -D warnings`, `cargo test --workspace`.
76+
- Commit style: `area: concise change` (e.g., `executor: fix buffer rollback`).
77+
78+
## Memory bank for agents (RAG)
79+
80+
We maintain a human‑readable “memory bank” for agents and humans under `/ai/memory` (Markdown + YAML frontmatter). Start with:
81+
82+
- `/ai/memory/index.md` — how to read the bank
83+
- `/ai/memory/architecture.md` — overview
84+
- `/ai/memory/components/` — component facts
85+
- `/ai/memory/decisions/` — ADR‑lite decisions
86+
- `/ai/memory/invariants.md` — project invariants
87+
88+
Agent workflow requirement: after you implement or change behavior, update the relevant files under `/ai/memory` (components, decisions, invariants, architecture) so future agents have accurate context.

‎ai/README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Codex Memory Model for This Repo
2+
3+
This repository adopts a human-readable memory model for agents and humans.
4+
5+
Structure:
6+
7+
```
8+
/ai/
9+
memory/ # invariants, decisions, architecture, components
10+
log/ # project log (optional, day-by-day)
11+
README.md # this file — how to use the memory bank
12+
```
13+
14+
Agent usage quickstart:
15+
16+
- Read `/ai/memory/index.md` first.
17+
- For architecture answers: load `architecture.md`, relevant component memos, and all invariants with `importance >= 0.8`.
18+
- For code generation: respect all invariants in `/ai/memory`.
19+
- If an invariant would be violated, state which one and propose an alternative.
20+
21+
See `/ai/memory/CODEX_MEMORY_MODEL.md` for the complete model description.
22+
23+
Agent workflow requirement:
24+
25+
- After implementing or changing behavior, update the corresponding files under `/ai/memory` (components, decisions, invariants, architecture). Keep the memory bank current so future agents have accurate context.

‎ai/log/.gitkeep‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎ai/memory/CODEX_MEMORY_MODEL.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
Below is a clean, compact memory dump for Codex/LLM agents that captures the key ideas of a “human‑readable memory bank” and “when to start a vector index.” Use as a system prompt, agent README, or memory-bank insert.
2+
3+
---
4+
5+
# 📦 Codex Memory Model — Dump
6+
7+
## 1. Big Picture
8+
9+
Project memory should be human‑readable, durable, and agent‑friendly.
10+
Primary format: Markdown + YAML frontmatter.
11+
Top directory:
12+
13+
```
14+
/ai/
15+
memory/ # invariants, ADR/decisions, architecture
16+
log/ # project diary
17+
README.md # how to use this
18+
```
19+
20+
---
21+
22+
## 2. Memory Record Format (Knowledge Fragments)
23+
24+
Every memory fragment is Markdown with a YAML frontmatter.
25+
26+
```markdown
27+
---
28+
id: inv-raft-001
29+
type: invariant # invariant | decision | fact | gotcha | todo | note
30+
scope: planner
31+
tags: ["raft", "sharding"]
32+
updated_at: "2025-11-29"
33+
importance: 0.95 # 0..1 — how critical it is to follow
34+
---
35+
36+
# Invariant: The Planner Operates Within a Single Raft Group
37+
38+
(human‑readable explanation)
39+
```
40+
41+
Why this format:
42+
43+
- readable as documentation,
44+
- easy to parse by agents,
45+
- YAML provides the structure,
46+
- Markdown provides narrative and context.
47+
48+
---
49+
50+
## 3. Memory Types
51+
52+
Use a small fixed set:
53+
54+
- `invariant` — a principle that must not be violated
55+
- `decision` — an architectural decision (ADR‑lite)
56+
- `fact` — important information about the system
57+
- `gotcha` — pitfalls and caveats
58+
- `todo` — long‑lived improvements, not tasks
59+
- `note` — useful observations
60+
61+
This helps agents:
62+
for architecture — look at `invariant` and `decision`,
63+
for debugging — `gotcha`,
64+
for system analysis — `fact`.
65+
66+
---
67+
68+
## 4. Separate Memory Files
69+
70+
Example layout:
71+
72+
```
73+
/ai/memory/index.md
74+
/ai/memory/invariants.md
75+
/ai/memory/architecture.md
76+
/ai/memory/components/planner.md
77+
/ai/memory/components/storage.md
78+
/ai/memory/decisions/0001-sharding-model.md
79+
/ai/memory/decisions/0002-datafusion-integration.md
80+
```
81+
82+
`index.md` describes structure and reading order.
83+
84+
---
85+
86+
## 5. Instructions for LLM Agents
87+
88+
Agents must:
89+
90+
1. Read `/ai/memory/index.md` first.
91+
2. Before architecture answers, load:
92+
- `architecture.md`
93+
- relevant components
94+
- all `invariant` with `importance >= 0.8`
95+
3. Before code generation, honor all invariants in `/ai/memory`.
96+
4. If an invariant would be violated, name it and propose an alternative.
97+
5. Avoid stale notes (`updated_at` too old or a `deprecated` flag, if present).
98+
99+
---
100+
101+
## 6. When to Start a Vector Index
102+
103+
Two layers of indexing:
104+
105+
### Layer A — Vector Index of Memory (`/ai/memory`)
106+
107+
Start early, after ~10–20 meaningful notes. Cheap, stable, useful.
108+
109+
### Layer B — Vector Index of Source Code
110+
111+
Start when 2–3 conditions hold:
112+
113+
- multiple subsystems/crates,
114+
- architecture relatively stabilized,
115+
- “where is X implemented?” requires significant search,
116+
- context does not fit a single prompt.
117+
118+
Initial scope only:
119+
120+
1. public APIs,
121+
2. key modules (planner/storage/executor),
122+
3. doc comments and gotchas.
123+
124+
Grow beyond this as the project scales.
125+
126+
---
127+
128+
## 7. Why Not Start Too Early
129+
130+
- code structure changes rapidly → index rots quickly;
131+
- noise outweighs signal;
132+
- maintenance cost > value early on;
133+
- while the project is small, LSP/grep is enough.
134+
135+
---
136+
137+
## 8. Practical Timeline
138+
139+
### Phase 1: first weeks
140+
141+
Create `/ai/memory/*.md` with no index yet.
142+
143+
### Phase 2: 10–20 memory fragments exist
144+
145+
Start the memory vector index (still not code).
146+
147+
### Phase 3: architecture stabilized, project grew
148+
149+
Start the source code index.
150+
151+
---
152+
153+
## 9. Project Log (Optional)
154+
155+
In `/ai/log/YYYY-MM-DD.md` keep human‑readable diaries:
156+
157+
- key decisions,
158+
- observations,
159+
- issues.
160+
161+
This is a RAG data source, but not part of invariants.
162+
163+
---
164+
165+
# ✔ Recommended Standard of Memory for Codex/LLM Agents
166+
167+
Optionally, we can:
168+
169+
- generate templates (memory template generator),
170+
- provide JSON Schema for frontmatter validation,
171+
- generate an example `/ai/memory` for your project (pg_fusion / picodata),
172+
- suggest an index refresh strategy (pre‑commit hook + partial refresh).

‎ai/memory/architecture.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
id: arch-overview-0001
3+
type: fact
4+
scope: repo
5+
tags: ["architecture", "datafusion", "pgrx", "shared-memory", "ipc"]
6+
updated_at: "2025-11-29"
7+
importance: 0.8
8+
---
9+
10+
# pg_fusion Architecture Overview
11+
12+
In short: a PostgreSQL (pgrx) extension intercepts planning/execution and delegates to a separate Apache DataFusion runtime. Communication uses shared memory (lock‑free rings + slot buffers for heap pages). The wire protocol is defined under `protocol/`.
13+
14+
## Top‑Level Directories
15+
16+
- `postgres/`: pgrx extension — plan/execute, IPC with runtime, builds Slot/MinimalTuple.
17+
- `executor/`: DataFusion runtime — parse/optimize/plan/execute, PgScanExec/Stream, encodes results into wire tuples.
18+
- `protocol/`: control and data messages, wire tuple/attribute formats, types.
19+
- `storage/`: low‑level heap page reader and attribute decoder to ScalarValue.
20+
- `common/`: shared errors/types (FusionError).
21+
22+
## Control Path
23+
24+
1. Parse → Metadata → Compile (logical plan)
25+
2. Bind (Columns) → Optimize → Translate (physical plan)
26+
3. BeginScan (register channels/slots) → ExecScan (start) → ExecReady
27+
4. EndScan (state reset)
28+
29+
## Data Path
30+
31+
- Executor requests heap blocks (scan_id, table_oid, slot_id).
32+
- Backend reads blocks, copies into SHM slots, sends metadata + visibility bitmap length.
33+
- `PgScanStream` reads pages from SHM, decodes tuples via `storage::heap`, builds Arrow RecordBatches.
34+
- Results are encoded to wire MinimalTuple and written to the result ring; backend reads frames and fills `TupleTableSlot`.
35+
36+
## Responsibilities
37+
38+
- Backend (`postgres/`): PG memory safety, `TupleTableSlot` formation, control FSM, heap IO.
39+
- Executor (`executor/`): DataFusion planning/execution, heap requests, decode/encode results, backpressure.
40+
- Protocol: stable binary formats/messages.
41+
- Storage: precise heap/attribute decoder (zero‑copy where possible).

‎ai/memory/components/.gitkeep‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎ai/memory/components/executor.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
id: comp-executor-0001
3+
type: fact
4+
scope: executor
5+
tags: ["datafusion", "runtime", "pgscan", "ipc", "arrow"]
6+
updated_at: "2025-11-29"
7+
importance: 0.8
8+
---
9+
10+
# Component: Executor (DataFusion Runtime)
11+
12+
## Essentials
13+
14+
- Table source: `PgTableProvider` → physical node `PgScanExec` → `PgScanStream`.
15+
- On `scan.execute()`, register the heap‑block receiver in a per‑connection `ScanRegistry`.
16+
- `PgScanStream` decodes heap pages (via `storage::heap`) to Arrow, then results are encoded as wire MinimalTuple and written to the result ring.
17+
- Partition strategy: temporarily force `target_partitions = 1` (see decision `dec-0004`).
18+
19+
## Public Surfaces
20+
21+
- `server::{parse, optimize, translate, start_data_flow, end_data_flow}` — control‑path FSM.
22+
- `pgscan::{PgTableProvider, PgScanExec, ScanRegistry}` — sources/scans.
23+
24+
## Gotchas
25+
26+
- JOIN in multi‑partition mode emits no rows without partition‑aware reading — use single partition.
27+
- SHM: borrowed slices must not outlive the producer’s write; do not cache references.

0 commit comments

Comments
 (0)