SaaS Metrics
Sorry for the pretentious name, I know, I know.. It just contains all the pieces I would like to see a AGI model to have, and I can't stand the temptation. Before throwing rocks at me, please take a glance at the Readme, and I hope it will cover your mood a little bit.So, first of all it does work and you can see the sample from the whole training run here: https://raw.githubusercontent.com/volotat/mini-AGI/refs/head...Here is the scaling law graph I have so far, and it looks very promising:
https://github.com/volotat/mini-AGI/blob/main/assets/scaling...The model was built under my deep dissatisfaction so we cannot really train even moderately big models (1B+ scale) on the consumer's hardware. We can inference and fine-tune them for sure, but I would like to have full control over what the model sees over the training run, so it is fully aligned with my interests, not some corporations.I was thinking about for some time and come up with two interesting ideas I thought worth pursuing: MoE with a lot of experts that gets added and pruned from the model while it trains, where only a small subset of of experts are actually in use at any particular moment + batch 1 training on the single continuous stream of data.First allows us to be bounded only by the disk space in terms of number of parameters and load and unload experts only when they are needed. The second (if figured out and it turns out to be doable) allows us to get aways with small VRAM capacity because we do not need to store big randomized batches and their respective gradients.I started brainstorming with Claude and after some time we found an approach that seems to be promising, and low and behold, a few weeks pass and you can see the results yourself.Obviously, I did use AI in the process of making this project and I am pretty sure it would be completely impossible for me to do something like this without it, so I hope it is more than justified.The model is still running over the first of 7.8B characters corpus I selected for training, so the weights are not out yet, and it's about a couple weeks of waiting until they are cooked at the current reading speed. And yeah, the model just read continuous interleaved passages from the dataset, each by 32K characters long each as a single stream. Just as you or I would do.The set up seems to be really simple so you can git clone the project, run it and observe everything for yourself.Thanks for your attention.
Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM
No tagline provided.
View Origin LinkProduct Positioning & Context
Related Ecosystem & Alternatives
Discover adjacent products, open-source repositories, and developer tools sharing similar technical architecture.
Deep-Dive FAQs
What is Mini-AGI – Dynamic continual learning model trained on 8GB VRAM?
Mini-AGI – Dynamic continual learning model trained on 8GB VRAM is a digital product or tool described as: Sorry for the pretentious name, I know, I know.. It just contains all the pieces I would like to see a AGI model to have, and I can't stand the tem...
Where did Mini-AGI – Dynamic continual learning model trained on 8GB VRAM originate?
Data for Mini-AGI – Dynamic continual learning model trained on 8GB VRAM was aggregated directly from the Hacker News community ecosystem, providing a stored public-discussion sample, not a representative customer survey.
When was Mini-AGI – Dynamic continual learning model trained on 8GB VRAM publicly launched?
The initial public indexing or launch date for Mini-AGI – Dynamic continual learning model trained on 8GB VRAM within our tracked developer communities was recorded on September 21, 2026.
How popular is Mini-AGI – Dynamic continual learning model trained on 8GB VRAM?
Mini-AGI – Dynamic continual learning model trained on 8GB VRAM has recorded public engagement of 277 traction score and facilitating 80 recorded discussions or engagements.
What are some commercial alternatives to Mini-AGI – Dynamic continual learning model trained on 8GB VRAM?
Text similarity retrieved a product for further investigation: Tobira Site Agent. Functional substitutability has not been verified.
How does the creator describe Mini-AGI – Dynamic continual learning model trained on 8GB VRAM?
The original author or development team describes the product as follows: "Sorry for the pretentious name, I know, I know.. It just contains all the pieces I would like to see a AGI model to have, and I can't stand the temptation. Before throwing rocks at me, please take ..."
Community Voice & Feedback
I built a smaller scale replication. Most of your claims prove out but the lever is essentially Biderman et als "learn less forget less". My first pass actually made this mistake initially and the findings didn't replicate. After controlling for step density I replicated your findings. Here is a write up:https://huggingface.co/spaces/dreddnafious/mini-agi-replicat...At the bottom of my write up I added some ideas to pin down the actual dominant parameters. I also added a pr to fix an issue in the codebase:PR: https://github.com/volotat/mini-AGI/pull/20What it fixes: a crash in upstream's GradSNR meter (issue #19), a diagnostic that tracks how much of the gradient is signal versus noise.
Mini-AGI is a totally inappropriate name - it seems what this project is shooting for, but not delivering on, is being a language model with "continual learning".Where it seems to fail, by design, on this goal is in delivering continual learning that is more than just "memorization with LRU catastrophic forgetting".That said, props to the author for thinking different and actually implementing something. Maybe the project can grow into something more, or inspire different ideas, if they continue to work on it.
This is an interesting approach. First of all, thanks for sharing your work. I've done. I want to say similar work in that I have trained continuous learning models and I have also offloaded parametric knowledge to hard drive people underestimate how difficult that is to do in a functional model. I look forward to digging in deeper.
I have not looked carefully but it seems like this is over-promising on avoiding catastrophic forgetting.The "trunk learning rate" is set at 0.1x the learning rate for the experts, so learning on different subjects disproportionately happens in the experts, and the trunk portion is comparatively more stable. But the population of experts can grow and shrink:> The pool grows when it is short of capacity and shrinks when parts of it stop being asked for.So:- doesn't the trunk then _eventually_ still undergo catastrophic forgetting, it just may take much longer?- and before that point, catastrophic forgetting happens in stepwise chunks whenever the expert pool shrinks?
Interesting. I wonder how much could be gained from using tokenization, which makes the model work at a semantic level rather than a syntactic level? I think it’s a force multiplier, but idk if it works here.
If you actually scroll through the transcript he links to, you will see that something that looks like it could be training is happening, but no coherent responses are coming out at any point. At least not that I saw skimming through.That might explain why there are no benchmarks of any kind.
This is the first thing I see in my life that really looks like proto-AGI, it deserves its name.
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
Nobody will throw rocks, I think most people are curious/suspicious about the big players and wants more hands-on since we suspect that this all will come down in cost soon enough.
Discovery Source
Hacker News Aggregated via automated community intelligence tracking.
Tech Stack Dependencies
No direct open-source NPM package mentions detected in the product documentation.
Media Tractions & Mentions
No mainstream media stories specifically mentioning this product name have been intercepted yet.
Deep Research & Science
No direct peer-reviewed scientific literature matched with this product's architecture.