MoonshotAI/Attention-Residuals

俺只是随手发了一个，大体就是，对残差链接做注意力也有文章，而且效果普遍都不是很好，却只字不提，之前的线性层那篇文章，也是和 RWKV 如出一辙，而且两篇一起看的话，作者明显是读过类似文章的

I’m a bit confused by the flow of this thread.

The OP originally linked to the "DeepCrossAttention paper" (published Feb 10, 2025). Since that paper's concepts seem very closely related to this repository's "Attention Residuals" (published Mar 16, 2026), the OP's concern about a missing citation feels completely valid. (similar concern on X: https://x.com/behrouz_ali/status/2033581834953453853 )

The response however seems to focus on comparing Attention Residuals to Hyper Connection, bypassing DeepCrossAttention. I checked the edit history, and it doesn't look like the OP ever changed their URL...

Was there a mix up when reading the initial issue?

啊？咱们下载的不是同一篇技术报告？

> https://arxiv.org/abs/2502.06785 和这篇几乎一样，但是文章中一点也不提及 之前也是这样 [MoonshotAI/Kimi-Linear#4](https://github.com/MoonshotAI/Kimi-Linear/issues/4)


Attention Residual是Layer Dimension的Quadratic Attention, Hyper Connection是Layer Dimension的Linear Attention. 这种关于layer维度的，之前有过一些 相关 工作，比如 [Learning Deep Transformer Models for Machine Translation ](https://arxiv.org/abs/1906.01787) , [Depth-Wise Attention (DWAtt)](https://arxiv.org/abs/2209.15168). 这种Attention Residual的工作相比于线性，可能在层数比较多(e.g. more than 1000)时候有一些明显效果提升相比于线性注意力(e.g. Hyper Connection)

Attention在层间流动的信息本质是在做gradient descent，Pre-Norm 和 Post-Norm是在流形上的优化，x是当前点，ffn(x)或者att(x)是对应的梯度，所以可以做很多有意思的东西。从这个角度理解 Attention Residual在做类似SGD+momentum。同时，我们也可以将信息流动建模成流形上的梯度下降 [GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization](https://arxiv.org/abs/2601.22095)

我对架构方向做的比较多，有问题可以多交流