Fable Makes An Audio Codec

I’ve been debating creating a blog for a while. Many of the projects I undertake don’t make good 5-minute spiels, and even if they do there are always more details to be shared. As I progress further into college, I feel it’s appropriate to coalesce my thoughts and meanderings into a cohesive website. I also plan to publish and link my research and open source projects through this website.

An LLM-related post is not the way I wanted to open my blog, but since AI is topical I suppose it’s fine. As a further bonus, this post was published on 11/11 and takes 11 minutes to read! Enjoy.

Background

I like coding. I like coding by hand. But over time, I learned I also like when agents do the coding part themselves.

Family and friends introduced me to “agentic software development” via ChatGPT Codex. I am old enough to remember the early days of AI, when you had to press ⌘-I to get an inline popup in VSCode, and ask the magical word machine to write a function or two. Sometimes, it would even do it correctly. But as time passed people realized that reviewing code was just not that much fun, so agents started writing and checking whole features all on their own.

Codex had generous limits at the time, and I used it extensively to create my first agentically developed software. As a music composition / computer science dual major, it was only natural that I’d try to make a Digital Audio Workstation (DAW). I named it Dawn. Dawn is written entirely in Rust, has a custom UI library and a carefully whiteboarded architecture designed to play to the strengths and weaknesses of LLM-based agents.

Dawn is a good candidate for agentically developed software. I am but one person with little free time, and DAWs are massive pieces of software undertaken by large teams at sizable companies. I don’t have the time or resources to match that, but I do have AI.

I wouldn’t call it vibe-coded in the slightest. For most of development, there were few vibes involved. Careful planning, decision making, and research made the software fast and stable. Dawn also contains an inter-process communication (IPC) library that I use to sandbox faulty plugins, and it’s faster than most IPC solutions out there [1]. Should I be expected to know the optimal way to transfer data between programs on every operating system?

As development progressed, Codex’s usage limits gradually tightened, and models became less intelligent. As a college student, it was looking more like Anthropic would be a better fit for my usage patterns: smarter models, less usage, and for the love of god: auto-continue. I’ve never felt more valued as a customer.

Fable 5.2 is released

Around the time I switched to Claude Code, Anthropic released Fable 5.2, and as a treat, granted limited availability to the measly $20 plan holders. I’ve never had use for such a model. My tasks generally have narrow scope and specific requirements, but one problem kept nagging at me.

You see, Dawn had a small issue with saving and loading projects. Recently, I was fortunate enough to sound design and mix an original stage musical. I had an entire 3-hour-long multitrack recording of the premiere to stress-test Dawn.

Most digital audio workstations store data in an uncompressed format for performance reasons. I tried FLAC first, and quickly discovered why. FLAC, the Free Lossless Audio Codec, is a state-of-the-art lossless audio compression library [2]. It achieves competitive compression ratios with superior performance, but it is simply not fast enough.

Saving the entire musical to disk with FLAC took an embarrassing number of seconds. This was a low-priority issue for me at the time. However, with my newly gifted Fable access… I couldn’t resist.

Quite literally, I told it:

“Come up with an audio compression codec that will be implemented in Rust… We’re targeting lossless compression, with an emphasis on speed, but it would be nice to approach or rival FLAC ratios 1-5 at faster speeds… You could try targeting your codec around more complex SIMD instructions that exist on most ARM and x86 platforms…”

And with that, we were off to the races. Fable returned a spec for a codec it named Riffle. Once Opus 5.5 implemented it, Riffle blew FLAC out of the water. To stress-test the codecs, I forced Dawn to fully cycle audio assets from memory to disk and vice-versa. The following are benchmarks on an M1 Max with 64 GB of RAM, using 24-bit audio:

Riffle nearly matches FLAC’s compression ratio, and does so while encoding faster than the uncompressed and unprocessed WAV files. This is because at these speeds, the disk is the limiting factor. Compressing and encoding our WAV file actually means there’s less disk traffic. In other words, doing nothing with our data is slower than doing something with it.

A more careful benchmark on a single musical audio file reveals some nuance on smaller workloads:

So how did Fable pull this off?

The Riffle Audio Codec

This section goes over the compression scheme of the codec, and why it works. If that doesn’t interest you, feel free to skip to the conclusion.

What I Know

I won’t pretend to know everything about how this was achieved. Frankly, I didn’t care enough about this project to begin with. I was trying to burn Fable credits gifted to me by Anthropic. But I suspect a couple of things went into making this one-shot prompt possible.

I knew that FLAC’s compression algorithm suffers from serial dependency chains. FLAC uses a mix of polynomial approximation and linear predictive coding [3] to guess at what the next sample of a waveform might be. This guess will likely be close to the correct answer, but rarely the correct answer. We can store the difference between our guesses and the answers alongside our prediction function.

If our predictor function is good, then the residual $𝑟\left[𝑛\right]$ will be a small number close to zero. Now, we can take advantage of Rice coding [4] to list the remainders in a near-optimal manner. In Rice coding, the smaller the number, the fewer bits are required to store it.

But the main problem here is parallelization and pipelining. Each Rice code has a different length, and FLAC writes them back to back. Where the code for sample $𝑛$ goes depends on where the code for sample $𝑛-1$ ended. To know that, we have to write code $𝑛-1$, and only then can we write code $𝑛$.

Modern CPUs have many registers and arithmetic units for computation, but the CPU can’t use them all because it has to wait until code $𝑛-1$ is written before moving on. It would be great if both the encoding and decoding processes were independent per sample, such that we could do work with whole chunks of samples at a time. In other words, we are latency-bound, not throughput-bound.

If this were possible, we could utilize wider SIMD registers [5] to operate on larger chunks of data. We can take advantage of this hardware capability to accelerate our task if we represent our compression algorithm in terms of this smaller subset of more powerful operations.

We need our math to conform to this paradigm, and we have a lot working in our favor. SIMD operations are more advanced than they used to be. In addition to add, subtract, and multiply, we now have swizzling, rotation, and shuffles. That brings us to my original prompt: Can Fable represent a more complex mathematical transformation using these more obtuse, blunt tools?

Fable’s answer

Riffle is a lossless audio format whose decoder has no sample-to-sample dependency, so every stage runs as SIMD over 8 to 32 samples at once. It keeps FLAC’s statistical model, linear prediction plus Rice codes, and changes the layout and the algebra so that model maps onto vector instructions.

The design rests on three ideas:

1. Split-plane Rice coding. Code lengths are identical to Rice codes. Remainder bits are bit-packed at fixed width, and quotients live in a separate unary bitmap. Every field’s offset is known from a header.
2. Feed-forward prediction. The encoder runs the recursive filter. The decoder runs a FIR filter over residuals, then running sums, which SIMD computes as a scan. An optional block-recurrent LPC stage reproduces FLAC’s predictor with one dependency per 4 samples instead of one per sample.
3. Ring arithmetic. All sample math is defined $mod{2}^{16}$, or $mod{2}^{32}$ for deep audio. Wraparound is the specification, so lanes never widen and nothing checks for overflow.

…

In plain English, Riffle attacks the two places where FLAC forces the CPU to work one sample at a time. The first is writing the data. FLAC’s compressed stream is like a sentence with no spaces: each number has a different length, so you can’t know where number $𝑛$ goes until you’ve finished writing number $𝑛-1$. Riffle splits every code into a fixed-size part and a variable-size part, and files them in two separate piles. The fixed-size parts of a group always take up the same amount of space, so simple arithmetic tells us where every value goes before we write a single one. It’s the same total size as FLAC’s codes, but now the CPU can pack 8 values at once.

The second is predicting the audio. Riffle’s simplest predictor is just how much each sample changed from the last one:

Unlike the decoder, the encoder already has every sample in hand, so no $𝑑\left[𝑛\right]$ has to wait on any other. Put 8 samples side by side in one SIMD register, say a ramp from 1 to 8. Shift a copy one slot to the right, and subtract it, all slots at once:

That’s 8 samples in a single step, and a wider register means more samples per step for free. The next group needs only one value from this one: its last sample. Higher-order predictors are just this done 2, 3, or 4 times over, so they get the same treatment.

Finally, all the math takes advantage of modular arithmetic. Computers do this by default: $\text{Max}+1=\text{Min}$, or if the number is unsigned: $\text{Max}+1=0$. Most codecs treat this as an error, so they do their math in wider registers so that these boundary cases are never reached. That can halve how many samples fit in each wide SIMD register, just for safety padding. Riffle treats the boundary conditions as part of the algorithm:

This means we can pack more data into a single register, and compute more values at a time.

Conclusions

So what does this all mean? Fable was able to, successfully, complete a task I didn’t expect it to make much progress on. I can see a couple of avenues of thought regarding this task and LLM capability in general.

1) The task was not hard, just different

Not having been done before doesn’t make something difficult. There are plenty of reasons why FLAC is slower than Riffle. FLAC was designed around 2001, when consumer SIMD was in its infancy and many of the tricks Riffle leans on didn’t exist yet. More importantly, FLAC’s format is frozen. Countless devices decode it, and changing the bitstream layout to suit modern vector instructions would break every one of them. FLAC chose compatibility over speed, which is a genuine consideration, but not for my use case.

2) It has been done before, sort of

There exists a codec called HALAC [6] with impressive benchmarks. But it only supports 16-bit audio, which is a deal-breaker for my use case. DAWs save audio in high-quality formats (usually 24-bit) because a lot of processing is applied to that audio. In a long chain of audio effects, a small change in the source audio can have a large impact on the final product.

Also, HALAC doesn’t seem to support Apple Silicon very well. I don’t imagine it’s supposed to be this slow on my benchmark.

On a single piece of musical 16-bit audio, FLAC (level 0) is now faster than Riffle (fast) on encoding.

Riffle remains superior as Dawn’s disk audio format. It decodes faster for streaming, and encodes at comparable speeds with ratios comparable to FLAC’s. In any case, 16-bit audio is not a primary use case for me; these benchmarks are for completeness’ sake.

Could I have come up with this?

I’m not sure. Probably, but it requires more than a conceptual overview of how FLAC works, down to how it fiddles with the bits. I’ve never read the FLAC source code, nor do I know exactly how they put the theory into practice. It would have required significant whiteboard time to sketch out the idea, try multiple ways of representing the problem, etc. Finally, I would land on a solution that jibes with both the math and the computer architecture. Worst of all, it requires you to “see” it: the one right option in the sea of permutations. This process is beneficial for cognitive development. But is it beneficial to my target goal?

This experiment was supposed to be an afterthought, but the results were interesting enough to write about, and tangibly useful in my application.

Large language models are smart. At least, they can be. Fable found relevant literature, synthesized new ideas from existing ones, and has results to show for it. If that’s not intelligence, what is? I don’t understand Riffle any more than I understand FLAC. But I knew enough to steer Fable in the right direction. We all deserve better software, and making better software is easier than ever. All it takes is for the operator to care.

But if AIs become more and more are capable, where does that leave us humans?