7. How memory systems handle long conversations without context explosion

The instinct is to keep the entire conversation in the prompt and trust the growing context window. That is the dump-the-transcript anti-pattern, and it fails for three compounding reasons: context windows are not free (cache misses on volatile content are expensive), more tokens often make the model worse (lost-in-the-middle, attention dilution), and transcripts are the wrong shape for retrieval.

Memory systems solve this with a fixed per-turn loop that reassembles the prompt from durable memory instead of accumulating raw transcript. The diagram below shows all six steps and where each AWS service fits.

Figure 7: The six-step per-turn loop. Step 3 is the key — the prompt is rebuilt from durable memory every turn. Raw transcript stays in S3 trace log for replay, never in the prompt.

Why the dump-the-transcript pattern fails:

• Context windows are not free. Cache misses on a million-token prompt with volatile content are expensive. Most production agent workloads have enough volatile content per turn that cache hit rates end up mediocre.
• More tokens often make the model worse. Lost-in-the-middle, attention dilution, and context rot all appear in production at long context lengths.
• Transcripts are the wrong shape for retrieval. You cannot ask a transcript "what does this user prefer?" and get a clean answer. You can only ask the summary you extracted from it.

Each turn runs a fixed loop:

8. Two retrieval paths, not one

This is the most important structural decision in the system, and the one most often missed. Memory retrieval has two modes that look similar from the outside but have completely different requirements underneath: known-scope lookup (give me everything that applies) and semantic discovery (find what's relevant to this query).

These two modes require different backends, different query patterns, and different guarantees. Running them as a single retrieval path conflates exact lookup with ranked search — and breaks both.

Figure 8: Two parallel retrieval paths. Path A runs against DynamoDB — exhaustive enumeration, no cutoff, no ranking. Path B runs against Aurora pgvector + OpenSearch — ranked, top-K, with score thresholds.

Understanding these two paths is essential:

Known-scope lookup. "Give me all active policies and preferences that apply to this agent for this turn." The query knows exactly what it wants. It is enumeration — every policy must be returned, no top-K cutoff, no ranking. Cheap, deterministic, runs every turn. Feeds the static prefix of the prompt.

Semantic discovery. "Find facts and episodes conceptually relevant to this user message." The query doesn't know what it wants until it sees what comes back. Ranking matters. Top-K matters. Score thresholds matter.

Path A: Known-scope lookup (startup injection)

SQL — Aurora PostgreSQL:

-- Path A: All policy and preference memory for this turn.
-- Exhaustive, no ranking, no top-K cutoff. Runs on EVERY turn.
SELECT 'policy'     AS kind,
       policy_key   AS key,
       policy_value AS value,
       version      AS meta
  FROM policy_memory
 WHERE tenant_id      = :tenant_id
   AND (effective_until IS NULL OR effective_until > CURRENT_TIMESTAMP)

UNION ALL

SELECT 'preference'  AS kind,
       pref_key      AS key,
       pref_value    AS value,
       source        AS meta
  FROM preference_memory
 WHERE tenant_id = :tenant_id
   AND user_id   = :user_id

ORDER BY kind, key;

-- Ranks across ALL tenants
WITH ranked AS (
  SELECT * FROM fact_memory
  ORDER BY embedding <=> :q
  LIMIT 100
)
SELECT * FROM ranked
WHERE tenant_id = :t; -- too late!

-- Filters by scope BEFORE ranking
SELECT * FROM fact_memory
WHERE tenant_id      = :tenant_id
  AND status         = 'active'
  AND superseded_by  IS NULL
ORDER BY embedding <=> :q
LIMIT 5;

9. Hybrid scoring and fused rank

Pure vector retrieval is a first draft. It works for conceptual queries like "find documents about this topic" but fails for precise recall like "what is this user's timezone preference". Hybrid retrieval solves this by combining two complementary signals: vector similarity for conceptual recall and lexical match for term-exact recall.

The challenge is fusing these two signals into a single ranked list. The diagram below shows the fusion algorithm: both retrieval paths run in parallel, scores are normalized, weighted (0.4 vector / 0.6 lexical), and combined with a threshold cutoff.

Figure 9: Hybrid fusion algorithm. Vector hits come from pgvector HNSW; lexical hits from PostgreSQL FTS. Scores are normalized, weighted, and combined. Results below the 0.4 threshold are discarded.

Why hybrid retrieval outperforms pure vector search:

Hybrid retrieval combines two signals: vector similarity (pgvector cosine distance via HNSW index) for conceptual recall and lexical match (PostgreSQL FTS via GIN index) for term-exact recall, fused into a single ranked list.

SQL — Aurora pgvector hybrid retrieval:

-- Path B: Hybrid retrieval on Aurora pgvector.
-- Scope filter BEFORE any vector ranking.
WITH vector_hits AS (
    SELECT fact_id, content, source_run_id, confidence,
           (embedding <=> :query_embedding) AS vec_dist
      FROM fact_memory
     WHERE tenant_id      = :tenant_id            -- scope FIRST
       AND status         = 'active'
       AND superseded_by  IS NULL
       AND (user_id  IS NULL OR user_id  = :user_id)
       AND (expires_at IS NULL OR expires_at > NOW())
     ORDER BY embedding <=> :query_embedding
     LIMIT 20
),
lexical_hits AS (
    SELECT fact_id, content, source_run_id, confidence,
           ts_rank(
               to_tsvector('english', content),
               plainto_tsquery('english', :query_text)
           ) AS lex_score
      FROM fact_memory
     WHERE tenant_id = :tenant_id                 -- scope FIRST
       AND status    = 'active'
       AND superseded_by IS NULL
       AND to_tsvector('english', content)
           @@ plainto_tsquery('english', :query_text)
     ORDER BY lex_score DESC
     LIMIT 20
),
max_lex AS (
    SELECT MAX(lex_score) AS max_val FROM lexical_hits
),
fused AS (
    SELECT
        COALESCE(v.fact_id,  l.fact_id)  AS fact_id,
        COALESCE(v.content,  l.content)  AS content,
        (
            COALESCE(1.0 / (1.0 + v.vec_dist), 0.0) * 0.4
            +
            COALESCE(l.lex_score / NULLIF(m.max_val, 0), 0.0) * 0.6
        ) AS fused_score
      FROM      vector_hits v
      FULL OUTER JOIN lexical_hits l USING (fact_id)
      CROSS JOIN max_lex m
)
SELECT fact_id, content, fused_score,
       CASE
           WHEN fused_score >= 0.7 THEN 'high'
           WHEN fused_score >= 0.5 THEN 'standard'
           ELSE                          'low'
       END AS relevance_tier
  FROM fused
 WHERE fused_score >= 0.4   -- drop long tail
 ORDER BY fused_score DESC
 LIMIT 5;

Why 0.4/0.6 and not 0.5/0.5? When both signals fire on the same row, the row is genuinely relevant — the lexical hit confirms the vector hit. When only the vector fires, it is plausibly relevant but should rank lower. A 0.6 weight on lexical signal reduces false positives in the result set by ~30% while preserving recall on strongly relevant rows. Threshold 0.4 eliminates the long tail of weakly-related vector results that hurt more than they help.

Cascade fallback for production availability

Python — retrieval fallback chain:

async def retrieve_with_fallback(query, tenant_id, user_id):
    try:
        results = await hybrid_search(query, tenant_id, user_id)
        if results: return results, "hybrid"
    except VectorIndexUnavailable:
        log_degraded("vector_index_down")

    try:
        results = await fts_search(query, tenant_id, user_id)
        if results: return results, "lexical_only"
    except FTSIndexUnavailable:
        log_degraded("fts_index_down")

    results = await like_search(query, tenant_id, user_id)
    return results, "like_fallback"
# Cascade: hybrid → vector-only → lexical-only → exact LIKE

10. Why the memory manager matters more than another vector store

Most teams ask "which vector database should we use?" before they know what the surrounding system needs to do. The vector store is a commodity — pgvector, OpenSearch, Pinecone, they all do roughly the same thing. The hard part is the coordination layer above it: the memory manager.

The memory manager sits between your agent and five separate storage systems. It routes writes, enforces scope, handles deduplication, manages supersession, and assembles the token-budgeted prompt. Without it, you have five disconnected stores and no system.

Figure 10: The memory manager as coordination layer. One interface for the agent; five storage systems underneath. Routing, scope enforcement, dedup, and prompt assembly all happen here.

What the memory manager does

• Route by type. Policy → DynamoDB. Fact → Aurora. Trace → S3. The agent sees one interface; the manager routes underneath.
• Enforce scope. Every query carries tenant_id and user_id. The manager injects these into every SQL WHERE clause and DynamoDB filter before the store sees the query.
• Deduplicate at write. SHA-256 hash on (tenant_id, user_id, content). If the fact already exists, increment a reconfirmed_count instead of creating a duplicate.
• Handle invalidation and supersession. When a new fact contradicts an old fact, the old record is not deleted — it is marked superseded_by = new_fact_id and excluded from reads via WHERE superseded_by IS NULL.
• Assemble the prompt. The prompt builder is part of the memory manager. It knows the token budget, the slot allocation for each memory type, and the static vs dynamic structure that feeds Bedrock prompt caching.

Python — Memory Manager interface:

class MemoryManager:
    def __init__(self, aurora_client, dynamo_client, s3_client,
                 opensearch_client, bedrock_client):
        self.aurora      = aurora_client
        self.dynamo      = dynamo_client
        self.s3          = s3_client
        self.opensearch  = opensearch_client
        self.bedrock     = bedrock_client

    async def retrieve(self, query: str, tenant_id: str,
                      user_id: str) -> MemoryBundle:
        # Path A: Exact-match retrieval — DynamoDB
        policies    = await self._get_policies(tenant_id)
        preferences = await self._get_preferences(tenant_id, user_id)

        # Path B: Hybrid retrieval — Aurora + OpenSearch
        facts    = await self._hybrid_search_facts(query, tenant_id, user_id)
        episodes = await self._hybrid_search_episodes(query, tenant_id, user_id)

        return MemoryBundle(policies, preferences, facts, episodes)

    async def promote(self, candidate: FactCandidate,
                     tenant_id: str, user_id: str, run_id: str):
        # SHA-256 dedup + Aurora UPSERT
        content_hash = hashlib.sha256(
            f"{tenant_id}:{user_id}:{candidate.content}".encode()
        ).hexdigest()

        await self.aurora.execute("""
            INSERT INTO fact_memory (
                fact_id, tenant_id, user_id, content, content_hash,
                embedding, source_run_id, status, created_at
            ) VALUES (
                gen_random_uuid(), :tenant_id, :user_id, :content,
                :content_hash, :embedding, :run_id, 'active', NOW()
            )
            ON CONFLICT (tenant_id, content_hash) DO UPDATE
            SET reconfirmed_count = fact_memory.reconfirmed_count + 1,
                updated_at = NOW()
        """, {
            'tenant_id': tenant_id, 'user_id': user_id,
            'content': candidate.content, 'content_hash': content_hash,
            'embedding': candidate.embedding, 'run_id': run_id
        })