Skip to main content

Is a Fixed-Size Hidden State Memory, or Compression?

· 7 min read
p4r4d0xb0x
p4r4d0xb0x
Rustacean, AI, OSS Enthusiast

cover placeholder

Calling a recurrent model’s hidden state “memory” is convenient, but taking the term literally causes us to miss an important distinction. A hidden state acts like memory in the sense that it passes previous input to the next computation. Usually, however, it does not retain past tokens in their original form. Within a fixed number of dimensions, it updates statistics and traces learned to be useful for the current objective. A more precise description is therefore an objective-specific compressed state.

This distinction is not wordplay. If some information must be recovered exactly, an external store or attention may be necessary; other information may be adequately represented by summarized dynamics alone. When reading S4, Mamba, RWKV, and Resona, we should therefore ask not “how much does it store?” but “what state transformation is learned for which query?”

Three meanings of state, memory, and compression​

Term explainer: hidden state

The internal representation held by a model at the current position in a sequence. It is used to process the next input, and in a recurrent architecture it is updated from the previous state and the new input.

Term explainer: compressed representation

A representation that is smaller than the original data while trying to preserve as much information as possible for a particular task. The training objective determines what is preserved.

A general recurrent update can be simplified as h_t = f(h_{t-1}, x_t). If the dimensionality of h_t does not grow with sequence length, then a long input is passing through a bottleneck of constant size. This bottleneck is both an advantage and a constraint. A small state makes inference memory and transfer volume predictable, but it can also cause different pasts to collide by mapping them near the same state. There is no guarantee that such collisions can be completely avoided in information-theoretic terms.

S4: why a fixed state is more than a simple summary​

S4 is research aimed at modeling long sequences efficiently through a structured state space model. The paper starts from continuous-time linear systems, addresses long-range dependencies, and proposes a computationally tractable sequence layer using HiPPO-family initialization and structured matrices.

What the source says: S4 presents an SSM layer for modeling long-range dependencies together with an efficient computational structure, and reports strong performance on several long-context benchmarks.

Interpretation: S4’s state should not be understood merely as a “summary of the most recent few tokens.” Its linear dynamics can be designed to respond to past inputs across multiple time scales. Even so, it differs from a database that stores the original text in a searchable form. The state has meaning only within a particular computational path and is not guaranteed to answer a request to retrieve an arbitrary sentence verbatim.

Mamba: remembering selectively and forgetting selectively​

Mamba introduces input-dependent selectivity into an SSM. In the language of the abstract, the model adjusts how much information it propagates or forgets according to the input content. This makes the hidden state not a passive compression buffer but a dynamic filter with a learned write-and-forget policy.

Here, “remembering” must be separated into the fact that data enters the state and the fact that it can later be reconstructed in exactly the desired form. Mamba’s ability to process long dependencies on a particular benchmark does not mean that every past token is preserved. What remains are features useful for the function demanded by the benchmark. This difference is central to separating recurrent state from searchable memory.

RWKV: combining the intuition of attention with recurrent execution​

The RWKV family explores structures that retain the intuition of weighted aggregation similar to attention while being executable recurrently. Public descriptions of the RWKV papers pursue parallel training together with efficient recurrent inference.

Source and interpretation: RWKV demonstrates the design possibility that a recurrent state can represent weighted accumulation over the past. But an accumulated state is not infinite memory. Within the summary space determined by the state dimension and update rule, the model retains information needed for its next prediction. Since multiple pasts may map to the same state, calling it a memory that guarantees exact retrieval would be an overstatement.

Resona: an expanded context requiring source verification​

Resona is a research name designated for inclusion in this group in connection with fixed states, iterative inference, or memory-efficient sequence processing. However, the evidence available in the current writing environment does not include the paper’s exact arXiv identifier and full abstract. We therefore do not assert specific performance numbers or detailed mechanisms for Resona.

In this article, the context provided by Resona is marked unverified. Writing that it “remembers longer” or “solves compression loss” based on the name alone would violate source-bounded writing principles. Claims should be expanded only after the exact paper link, abstract, and experimental conditions have been checked.

External-memory context: Qdrant and Statey are a different layer​

Qdrant is a vector database that stores vectors and provides similarity search. As confirmed in the earlier article, Statey describes itself as a shared database that reads and writes structured records within a chat. Neither is the same kind of “internal state” as a recurrent hidden state.

Context confirmed for Qdrant and Statey: External stores do not leave data only within the fixed dimensionality of a session or model; they make it possible to search later or read it again as structured records. Qdrant emphasizes embedding similarity search, while Statey emphasizes MCP-based record sharing through different interfaces.

Interpretation: External memory increases storage capacity and retrievability, but it requires operations for write permissions, search quality, latency, data isolation, and deletion policy. A hidden state, by contrast, is passed very quickly during request processing, but it disappears when the session ends and is difficult to audit or recover without separate storage. Even if both are called “memory,” their trust boundaries and failure modes differ.

What belongs where in practice?​

  • Hidden state: Summaries of recent context, continuous signals, and state tracking for streaming input—information that needs to be updated at every token.
  • Search memory: Facts, documents, and user preferences for which the original text or supporting evidence must be retrieved later. Search keys and permissions must be designed together.
  • Structured records: Operational data such as state, ownership, and change history that requires exact fields and auditing.

This separation is not only a model-performance issue. Compression loss in a hidden state is reflected in the task loss, whereas external memory can inject incorrect writes or poisoned search results into the model. In other words, compression is a representational risk, while external memory is a data-governance risk.

Conclusion and limitations​

A fixed-size hidden state is used like memory, but in essence it is closer to a learned, objective-specific compressed state. S4 shows long-range dynamics, Mamba selective retention and forgetting, and RWKV a combination of recurrent execution and weighted aggregation. They all share the property of transforming the past into a bounded representation, but none promises exact storage or arbitrary search. Specific claims about Resona remain unverified until its identifier and abstract are checked.

Qdrant and Statey are examples of a separate layer: external memory. When using internal model state together with an external store, it must be explicit what is a summary and what is the original, who can read and write it, and how deletion and recovery work.

Sources​

// COMMENTS

Comments