Comment on: Talk is cheap. Show me the code.
Repo: MoonshotAI/Attention-Residuals by ddofer
I also implemented it for encoders. Performance was terrible if not compiling, and bad with compiling. (ProteinJEPA).
I may have missed something tho (e.g. custom kernels?)
GitHub Issue
SaaS Metrics