SaaS Metrics
Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
A novel approach to file compression that achieves high ratios (e.g., 100MB CSV to 7MB) by training a small transformer to 'memorize a single file and predict the next byte.'
Product Positioning & Context
AI Executive Synthesis
A novel approach to file compression that achieves high ratios (e.g., 100MB CSV to 7MB) by training a small transformer to 'memorize a single file and predict the next byte.'
This experiment demonstrates a specialized, high-ratio file compression technique with significant B2B potential, despite its current performance limitations. The ability to compress a 100MB CSV to 7MB is compelling for industries dealing with large, repetitive datasets, such as financial logs, sensor data, or specific database backups. While slow for general-purpose use, this method could be valuable for archival storage, cold data tiers, or specialized data transfer scenarios where compression ratio outweighs speed. The concept of a '900KB transformer' memorizing a single file suggests a highly optimized, domain-specific compression model. Enterprises could leverage this for reducing storage costs, improving data transfer efficiency for specific data types, or enhancing the performance of data lakes by minimizing I/O. Further optimization of training and inference speed would unlock broader applicability in enterprise data management.
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files.Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output.On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte).It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my AMD 7800XT).Checkout the repo - https://github.com/samyak112/pym-particles
Related Ecosystem & Alternatives
Discover adjacent products, open-source repositories, and developer tools sharing similar technical architecture.
Deep-Dive FAQs
What is Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB?
Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB is analyzed by our AI as: A novel approach to file compression that achieves high ratios (e.g., 100MB CSV to 7MB) by training a small transformer to 'memorize a single file and predict the next byte.'. It focuses on This experiment demonstrates a specialized, high-ratio file compression technique with significant B2B potential, despite its current performance l...
Where did Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB originate?
Data for Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB was aggregated directly from the Hacker News community ecosystem, representing raw developer and early-adopter sentiment.
When was Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB publicly launched?
The initial public indexing or launch date for Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB within our tracked developer communities was recorded on June 27, 2026.
How popular is Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB?
Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB has achieved measurable traction, logging over 98 traction score and facilitating 60 recorded discussions or engagements.
Which technical categories define Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB?
Based on metadata extraction, Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB is categorized under topics such as: overfitted transformer, arithmetic coding, compress individual files, predict the next byte.
What are some commercial alternatives to Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB?
Our semantic intelligence engine identifies potential commercial alternatives in the SaaS space, such as Navige, which offers overlapping value propositions.
How does the creator describe Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB?
The original author or development team describes the product as follows: "I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files.Instead of training the model to generalize, I train a 900KB transformer to memorize a s..."
Community Voice & Feedback
Great work. Just Yesterday I thought about LLMzip and asked myself if this is something which could vastly improve HTML compression when done at Google scale and shipped with browsers. I haven't done any research though.
Somewhat related is stavros's method to compress 500KB to something like 50 bytes https://www.stavros.io/posts/compressing-images-with-stable-...main drawback is that it's not lossless ;-)but this is great. I hope this actually becomes a format that wraps the weights and transformer module (maybe this can also be NAS-optimized too?). Maybe it would even work for video?It's like calling gzip but instead of compression level you choose kolmogorov complexity level
The model is the important part, a huffman code or adaptive huffman or other sorts of encoders would be much better on a dataset based on the model. You need the model to also decode. And on a dataset of sufficient size, embedding the model and the benefit of it's memorization of the file can be offset.A non-general compression algorithm (model - I don't mean a distinct llm, but "modeling data") targeted at a specific dataset will always do better than a general algorithm.The reason I mentioned the "encoder" doesn't matter - arithmetic coding, for the data it is presented, will beat huffman/adaptive huffman every day, but it's the model that is where the real "compression" comes into play.I've implemented enough "coders" over the years, including arithmetic for both commercial and research purposes (was a student of Glen Langdon).
Dumb question: can you train a model to predict the next byte of ANOTHER MODELSo apply this same logic to compressing a bigger model within a smaller modelI know this is absolutely regarded, but humour me please
I've had this idea of building a codec that would similarly overfit to specific images. But the codec itself would not be a fixed size transformer... instead you could just mess around with the sizing to get better quality/smaller size.So the codec would be something like:
I've seen experiments where people have a "fixed" pipeline but I think having something more dynamic would work quite well.
I've seen experiments where people have a "fixed" pipeline but I think having something more dynamic would work quite well.
Three questions:1. How much was AI used to generate documentation for this project?2. The 100MB CSV data sources are not provided in the repo so it doesn't seem possible to reproduce your results. The enwik9 dataset says it is a "slice" of the larger data set, and there are many NYC taxi trip record datasets that exist. Can you provide the datasets used to generate your results?3. I am surprised to see performance comparisons only between your transformer and WinZIP. What were your results when comparing your transformer to more modern approaches like LZMA2 (level 9), BZIP2 and ZPAQ (max effort)?
What do those compress to with conventional approaches? For comparison.I am curious. A classic machine learning ensemble approach is to overfit a collection of small models then bag them (e.g. voting) allowing the models to generalize.I'm sure someone's tried to overfit a bunch of transformers for compression like this, then bag them to see how well it does?
Fabrice Bellard may have been the first to do this, 7 years ago: https://news.ycombinator.com/item?id=27244004
Neat approach.
Since the 900KB model ships with the compressed file, is there a file size below which the model overhead just eats the gains?
Curious where the crossover is.
Since the 900KB model ships with the compressed file, is there a file size below which the model overhead just eats the gains?
Curious where the crossover is.
What does it compress the full 1GB file to? http://prize.hutter1.net/
Discovery Source
Hacker News Aggregated via automated community intelligence tracking.
Tech Stack Dependencies
No direct open-source NPM package mentions detected in the product documentation.
Media Tractions & Mentions
No mainstream media stories specifically mentioning this product name have been intercepted yet.
Deep Research & Science
No direct peer-reviewed scientific literature matched with this product's architecture.