Decayfmt: The File Format That Corrupts Itself as a Feature, Not a Bug

What Is Decayfmt? The Core Premise

Decayfmt is not a buggy parser. It is a deliberate inversion of the durability contract we expect from file systems. Created by Arav Panwar, the format guarantees that every time you open a file, the data degrades slightly—until it eventually becomes unreadable. Think of it as a digital half-life for information.

The source implementation lives at github.com/aravpanwar/decayfmt. It is a Python library that wraps a custom binary format. When you call open(), the library doesn't just read bytes; it mutates them according to a deterministic decay function. Close the file, and the corruption is permanent. Open it ten times, and you might have static. Open it a hundred times, and you have an empty husk.

This is not encryption with an expiration date. It is not a self-destructing message. It is bit-rot accelerated to human timescales. For engineers who spend their days fighting entropy with checksums, replication, and parity bits, Decayfmt is a philosophical gut punch: what if we stopped fighting and started designing with decay as a first-class citizen?

The Engineering Guts: A State-Machine of Rot

Peeking under the hood reveals a surprisingly clean architecture. The format is a finite state machine where the number of reads determines the state, and the state determines how much of the payload survives.

The flow is straightforward but brutal:

The format header stores a 32-bit unsigned integer representing the read count. The payload follows the header. On every open(), the library:

1. Reads the header to get the current count.
2. Increments the count.
3. Writes the new count back to the header immediately.
4. Uses the new count as a seed for a pseudo-random number generator (PRNG).
5. Flips bits in the payload based on the PRNG output.
6. Returns the mutated payload to the caller.

# Simplified from the decayfmt source
def open(self, filepath):
    with open(filepath, 'r+b') as f:
        count = int.from_bytes(f.read(4), 'big')
        count += 1
        f.seek(0)
        f.write(count.to_bytes(4, 'big'))
        payload = bytearray(f.read())
        random.seed(count)
        for i in range(len(payload)):
            if random.random() < DECAY_RATE:
                byte_index = random.randint(0, len(payload) - 1)
                bit_index = random.randint(0, 7)
                payload[byte_index] ^= (1 << bit_index)
        return bytes(payload)

The corruption function is deterministic given the read count. This means two identical files opened the same number of times will degrade in the exact same way. It also means you can pre-compute the state of a file after N reads without actually performing the reads, though the library doesn't expose this.

The decay rate is configurable. A low rate means the file survives hundreds of reads before becoming gibberish. A high rate means three reads and you're staring at noise. The default sits around 0.1% bit-flip probability per byte per read, which gives a roughly logarithmic degradation curve.

Why This Matters for Forward Deployed Engineers

Forward Deployed Engineers (FDEs) operate in the messy gap between prototype and production. They ship code into customer environments where data gravity, compliance, and legacy systems collide. Decayfmt might seem like a toy, but its design pattern maps directly onto real problems FDEs face daily.

Ephemeral Prototyping Without Cleanup

When you deploy a prototype into a customer's staging environment, you often leave behind test files, configuration stubs, and sample data. Standard practice is to write a cleanup script or rely on TTL-based object storage. Decayfmt offers a different model: the files clean themselves up through natural use. If a customer's team opens that sample CSV three times to inspect it, it corrupts and becomes useless, signaling that it was temporary.

This matters because cleanup scripts fail silently. They miss edge cases. They require maintenance. A self-decaying file is a self-documenting expiration policy. The file's own state communicates its freshness.

Forcing Data Hygiene in Integration Testing

Integration tests often use static fixtures that grow stale but never break. An FDE setting up a CI pipeline for a customer could use Decayfmt for test fixtures that must be regenerated regularly. If a fixture file has been opened more than five times, the tests start failing—not because the code is wrong, but because the data is too old. This forces the team to refresh their test data, preventing the "works on my machine" problem where stale fixtures mask integration drift.

This pattern aligns with the philosophy we explore in The Highest-Leverage Skills for an FDE in the AI Era: Speed, Taste, and Data Wrangling. Data wrangling isn't just about cleaning data; it's about building systems that enforce data freshness as a structural property. Decayfmt is a blunt but effective tool in that toolbox.

Anti-Caching for LLM Context Windows

FDEs building RAG pipelines or prompt chains often cache intermediate results to save on LLM API costs. But context changes. A cached summary of a customer's database schema from three months ago is worse than useless—it's misleading. Decayfmt could store those cached embeddings or summaries, ensuring that if you read the cache too many times without refreshing it, the data degrades and forces a fresh LLM call.

This is a form of cache invalidation that doesn't require a separate invalidation service, timestamp checks, or distributed consensus. The invalidation is embedded in the data itself. For an FDE shipping a prototype agent in a week, this eliminates an entire class of infrastructure concerns.

How to Try Decayfmt Today

The library is pure Python with no dependencies beyond the standard library. You can install it directly from the repository:

git clone https://github.com/aravpanwar/decayfmt.git
cd decayfmt
pip install -e .

Or just copy the core module into your project—it's a single file under 200 lines. Here is a minimal usage example:

from decayfmt import DecayFile

# Write a file that will decay over reads
df = DecayFile('ephemeral.dat', decay_rate=0.001)
df.write(b"This message will self-destruct, slowly.")

# First read: nearly pristine
data1 = df.read()
print(data1)  # b"This message will self-destruct, slowly."

# Tenth read: starting to show artifacts
for _ in range(9):
    data10 = df.read()
print(data10)  # b"This messa~e will self-destruct, slowly."

# Hundredth read: mostly noise
for _ in range(90):
    data100 = df.read()
print(data100)  # b"\x1f\xa3\xbb..."

Building an Ephemeral Logging System

A practical FDE use case: customer deployments often need verbose logging during initial rollout, but those logs shouldn't persist forever. Wrap your log writes in a Decayfmt file, and the logs naturally degrade as engineers inspect them during debugging.

import json
from decayfmt import DecayFile

class DecayingLogger:
    def __init__(self, path, decay_rate=0.0005):
        self.file = DecayFile(path, decay_rate)
        self.file.write(b"[]")  # Initialize empty JSON array

    def log(self, entry):
        data = json.loads(self.file.read())
        data.append(entry)
        self.file.write(json.dumps(data).encode())

    def tail(self, n=10):
        data = json.loads(self.file.read())
        return data[-n:]

Each tail() call increments the read counter and corrupts the log slightly. After a few dozen debugging sessions, the oldest entries become unreadable, naturally pruning the log without any explicit retention policy.

Integrating with Automated Workflows

If you're building agents that interact with file systems, as covered in Build a Multi-Agent Research Assistant with LangGraph and Groq Free Tier, you can use Decayfmt to manage intermediate artifacts. An agent that writes a research brief to disk can wrap it in a decaying format so that stale briefs don't accidentally get surfaced to users days later.

The Tradeoffs: Ephemerality as a Hard Constraint

Decayfmt is not a general-purpose format. It is a specialized tool for a specific class of problems. Understanding the tradeoffs is critical before you introduce it into any system, even a prototype.

Predictable Corruption Is Still Corruption

The deterministic nature of the corruption function is a double-edged sword. An attacker who knows the read count can reconstruct the original payload by reversing the bit flips. This is not encryption. Do not use Decayfmt for secrets, PII, or anything requiring confidentiality. The decay is a feature for data lifecycle management, not a security boundary.

No Partial Recovery

Once bits flip, they stay flipped. There is no parity, no error correction, no redundancy. If you need to recover data after it has degraded, you are out of luck. This is by design, but it means you must have a separate source of truth. Decayfmt files are caches, not primary stores.

Filesystem Assumptions

The library assumes it can read and write the same file in place. Network filesystems, object storage mounted via FUSE, or any system with eventual consistency semantics will break the atomic read-increment-write pattern. The counter could diverge, causing unpredictable corruption. Use Decayfmt only on local storage with POSIX semantics.

The Philosophical Tension

There is a deeper tradeoff that engineers feel viscerally. We are trained to preserve data. Checksums, replication, backups, WAL logs—our entire discipline is a war against bit rot. Decayfmt asks us to accept decay as a design material. This is uncomfortable. It goes against muscle memory.

But discomfort is where interesting engineering happens. The pattern of self-degrading data is not new—it appears in nature, in human memory, in analog media. Digital systems have been the outlier, insisting on perfect fidelity. Decayfmt is a small experiment in bringing digital systems closer to how information behaves in the physical world.

For FDEs, who operate at the boundary between clean abstractions and messy reality, this mindset is valuable. Customers don't have perfect data. Their databases have nulls where there should be foreign keys. Their logs have gaps. Building systems that degrade gracefully rather than failing catastrophically is a core skill. Decayfmt is a training wheel for that muscle.

If you're thinking about how to demonstrate this kind of systems thinking in your portfolio, The FDE Portfolio: 4 Projects to Build to Prove You Can Ship in the Customer's Chaos covers projects that show you can design for failure modes rather than pretending they don't exist.

FAQ

Is Decayfmt a joke or a serious tool? It is both. The implementation is a toy, but the design pattern is serious. Self-degrading data structures have real applications in cache invalidation, ephemeral logging, and test fixture management. The library is a provocation to think about decay as a feature.

Can I control the rate of decay? Yes. The decay_rate parameter controls the probability of a bit flip per byte per read. Values around 0.001 give gradual degradation over hundreds of reads. Values around 0.1 give aggressive degradation within a handful of reads.

Does the file eventually become completely empty? Not empty, but statistically indistinguishable from random noise. After enough reads, the bit flips will have touched every byte multiple times, resulting in maximum entropy. The file size remains constant.

Can I use this in production? Not in its current form. The single-file, no-dependency implementation is not production-hardened. But the pattern can be adapted. For example, you could implement a decaying cache layer in Redis by attaching a Lua script that mutates values on read, using the access count as a seed.

What happens if I copy a Decayfmt file? The copy inherits the current read count and the current level of corruption. It will continue degrading from that point. Copying does not reset the counter. If you need to preserve a snapshot, read the file once and write the output to a standard format.

Is there any way to reset the decay? No. The only way to get a pristine file is to create a new one from the original source data. This is by design. The irreversibility is what makes the decay meaningful.

How does this compare to TTL-based expiration? TTL is time-based; Decayfmt is access-based. A file that nobody opens doesn't decay. A file that is opened frequently decays quickly. This access-pattern-sensitive degradation is different from wall-clock expiration and can be more useful for debugging artifacts that should degrade as they are inspected.