A Machine That Sleeps: Rethinking Memory and Learning with Hyperdimensional Computing#

An engineering exploration of predictive memory, continual learning, and deliberate forgetting.

The problem with remembering everything#

I've been thinking about a question for a while: can we build a system that improves through experience without constantly retraining a large neural network?

Most AI memory systems I've worked with follow a familiar pattern. Take an interaction, turn it into an embedding, put it in a database, and retrieve it when something similar comes up.

That's useful, but retrieval isn't necessarily learning.

A system might remember that something happened a hundred times without becoming any better at predicting the hundred-and-first occurrence. Worse, storing everything creates a second problem: eventually the system has so much memory that deciding what matters becomes expensive.

I wanted to investigate a different architecture.

Instead of asking a model to remember everything, what if the system tried to predict what would happen next, saved experiences that exposed gaps in its understanding, and periodically reorganized its memories?

The working loop is:

Observe → Encode → Predict → Compare → Learn → Consolidate

The consolidation step is what I informally call sleep.

This is a design proposal, not a demonstrated intelligence system. The goal is to make each component small enough to implement, measure, and potentially disprove.

1. Why hyperdimensional computing?#

Hyperdimensional computing (HDC), also called vector symbolic computing, represents information using high-dimensional vectors called hypervectors.

Depending on the representation, a hypervector might contain thousands of binary or bipolar components.

HDC provides a few useful operations:

• Binding: Associate two representations, such as an object and its role.
• Bundling: Combine multiple representations into one that retains similarity to its constituents.
• Permutation: Encode order or position by transforming vector coordinates.
• Similarity: Compare representations, often using cosine similarity or Hamming distance.

For a binary implementation, binding can be an XOR operation, bundling can use majority voting, and similarity can be evaluated using bitwise operations.

That makes HDC attractive for a constrained experimental system. Much of the basic computation is predictable, parallelizable, and relatively simple to implement.

But simple computation doesn't automatically imply useful intelligence.

The challenge is what information enters the vectors and how experience changes the system's future behavior.

2. The encoder might be the hardest part#

Suppose the system sees two slightly different images of the same object.

If the encoder maps them to unrelated hypervectors, no clever memory algorithm will reliably recover the relationship.

This is where I think a lot of the difficulty lives.

My initial design separates the observation encoder from the memory system.

An observation at time t is transformed into a hypervector:

h(t) = E(x(t))

The encoder should preserve relationships relevant to the task, not merely produce uniformly distributed random vectors.

For categorical observations, deterministic item vectors and role binding may be sufficient. For continuous features, I'd start with quantization and similarity-preserving level encodings, then compare them with random projections.

Raw images, text, or audio present a much harder problem. I'd avoid starting with them until the learning loop works on structured observations.

This separation matters because I want to distinguish two failures:

1. The memory system cannot learn useful transitions.
2. The encoder has already discarded the information needed to learn them.

Without that distinction, it's easy to spend weeks optimizing the wrong component.

3. Learning through prediction error#

My proposed system maintains an encoded context containing the current observation and a bounded amount of recent history.

Given context and an action, it searches associative memory for similar prior situations and predicts the next encoded observation.

Conceptually:

Context + Action → Predicted Next Observation

When the real observation arrives, the system measures the difference between the prediction and what happened.

For binary hypervectors, one possible normalized error is:

Error = HammingDistance(predicted, actual) / dimension

This gives a value between zero and one. It is not a calibrated probability, but it provides an initial measure of representational disagreement.

The next question is when to update memory.

My first version would prioritize experiences where:

• Prediction error is high.
• The observation contains genuinely new information.
• The experience is likely to matter again.
• The observation is not simply noise or a duplicate.

A naive implementation would store every high-error event.

That would be a mistake.

An unpredictable random number generator would produce an endless stream of surprising observations. Memorizing those outcomes wouldn't improve future predictions.

So prediction error cannot be the only learning signal.

I'd combine surprise with novelty, recurrence, and evidence that similar contexts produce learnable outcomes. The admission policy needs a finite memory budget from day one.

4. Memory should be selective, not infinite#

I want two different forms of memory.

Episodic memory stores specific observations and transitions. It is useful for preserving unusual experiences and explaining why a prediction was made.

Consolidated memory stores repeated patterns that can be represented more compactly.

For example, if a system repeatedly observes that a particular action in a particular context produces the same outcome, it shouldn't need hundreds of nearly identical records to make that prediction.

A prototype could maintain a representative context, an outcome, its observation count, and evidence of how consistently the transition occurred.

However, blindly merging similar experiences can destroy important distinctions.

Two nearly identical situations might have different outcomes because of a hidden variable. Combining them into a single prototype could make the system confidently wrong.

I'd therefore preserve competing outcomes and uncertainty rather than always forcing them into one representative vector.

Memory management becomes an optimization problem:

Retain information that improves future predictions, subject to a bounded storage and computation budget.

This also creates a reason to forget.

Not simply because an experience is old, but because it is redundant, misleading, or no longer useful under the current environment.

5. What happens when the machine sleeps?#

The sleep phase would be a separate, offline consolidation process.

It would not generate dreams or simulate human consciousness.

Instead, it would examine accumulated experiences and look for opportunities to improve the memory structure.

The proposed procedure:

1. Identify repeated and highly similar transitions.
2. Merge compatible memories into supported prototypes.
3. Preserve exceptions and transitions with conflicting outcomes.
4. Decay redundant or low-value memories.
5. Evaluate the candidate memory against a held-out replay set.
6. Commit the updated memory only if it passes the evaluation.

The last two steps are particularly important.

Consolidation should be treated as a versioned operation, not a destructive cleanup task.

If compression damages prediction quality, the previous memory remains available.

This is similar to how I think about reliable AI workflows generally: a system proposing a change and a system accepting a change should not be the same decision boundary.

6. Do we actually need a GPU?#

My instinct is to start on the CPU.

If we're using 8,192-bit binary hypervectors, one vector occupies 1,024 bytes before metadata.

At 100,000 stored transitions, keeping both a key and an outcome vector would require approximately 195 MiB just for the packed vector payloads.

That's manageable on a typical laptop, but scanning a growing memory on every observation could become expensive.

Binary similarity can be computed using XOR followed by population count. Modern CPUs can perform these operations efficiently using packed words and vectorized instructions.

A GPU becomes attractive when the workload has enough parallel similarity comparisons or batch encoding operations to amortize dispatch and transfer overhead.

But moving data to a GPU isn't free.

I'd compare:

• A straightforward CPU implementation.
• A bit-packed, vectorized CPU implementation.
• A batched GPU implementation, if profiling justifies one.

The measurements should include end-to-end prediction latency, throughput, memory usage, and energy where it can be measured reliably.

I'm particularly interested in whether selective memory admission and consolidation can reduce the amount of computation needed in the first place.

Avoiding unnecessary work is often more valuable than accelerating it.

7. How I would test the idea#

I wouldn't start with a chatbot or claim the system can reason.

I'd begin with three controlled environments.

Experiment A: Learnable sequences#

Generate streams with deterministic transition patterns, inject controlled noise, and test whether the system learns to predict the next event.

This isolates encoding, temporal memory, and prediction.

Experiment B: Changing environments#

Introduce a regime change after the system has learned an earlier pattern.

Measure how quickly predictions adapt and whether obsolete memories interfere with learning.

This tests forgetting and adaptation.

Experiment C: Partially observable navigation#

Use a small environment in which actions affect future observations, but the current observation alone is insufficient to determine the state.

This tests whether temporal context and episodic recall provide a measurable advantage.

For each experiment, I'd compare the system with simpler baselines such as a bounded nearest-neighbor episodic store, an n-gram transition model, and a version without consolidation.

The key measurements would be prediction accuracy, adaptation time, memory footprint, retrieval latency, and the effect of consolidation on previously learned behaviors.

One concrete hypothesis is that consolidation can reduce stored memory by at least 30% while limiting relative prediction-performance degradation to 5% on a predefined test suite.

That's a target to test, not a measured result.

If a much simpler transition table consistently performs better, that is valuable evidence against the proposed complexity.

8. What might fail?#

Several things.

First, an inexpensive encoder may not preserve enough information for useful generalization.

Second, similarity in hypervector space does not necessarily imply similarity in outcomes.

Third, prediction error may prioritize noise instead of learnable information.

Fourth, consolidation might destroy rare but important exceptions.

Finally, even if the system learns structured environments effectively, that doesn't establish anything close to general intelligence.

Those limitations are part of why I find the experiment interesting.

The right first question isn't whether HDC can replace large language models.

It's whether a deliberately small architecture can develop useful predictive memory from experience, under explicit constraints, without continuously retraining a large model.

9. What I would build first#

The first implementation doesn't need a sophisticated interface or a distributed system.

I'd start with a small Rust or C++ library exposing five operations:

• encode(observation) to produce a deterministic hypervector.
• predict(context, action) to retrieve and combine candidate transitions.
• observe(context, action, outcome) to evaluate prediction error and update memory.
• consolidate() to construct a candidate compressed memory.
• evaluate(dataset) to compare the candidate against a reproducible baseline.

Memory would initially live in a contiguous, bit-packed representation.

I'd want reproducible random seeds, deterministic tests, and a small benchmark harness before introducing concurrency or GPU acceleration.

One important architectural decision would be keeping consolidation out of the online inference path.

Prediction should read an immutable memory snapshot. Consolidation can build a replacement snapshot and publish it only after validation.

That avoids making a partially completed consolidation visible to an active prediction loop.

It also makes rollback straightforward.

The interesting engineering work would begin when the memory budget is exhausted. At that point, every new experience must compete with existing information for storage.

That's the constraint I want to study.

Closing thoughts#

The idea I'm exploring is that memory should be an active part of learning, not just a searchable record of the past.

A useful system should notice what it doesn't understand, retain experiences that might improve future behavior, recognize repeated patterns, and discard information that no longer earns its place.

I don't know yet whether hyperdimensional computing is the best substrate for this.

But the architecture is small enough to build incrementally, and the hypotheses are concrete enough to test.

That's where I'd start.

---

Further reading#

• Pentti Kanerva, Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors (2009). https://doi.org/10.1007/s12559-009-9009-8
• Ali Rahimi and Benjamin Recht, Random Features for Large-Scale Kernel Machines (2007). https://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines