SaaS Metrics
Comment on: 請問有實現的程式碼嗎?
Repo: MoonshotAI/Attention-Residuals by ILoveSorasakiHina
```python
"""
Attention Residuals (AttnRes) — Full 版本實現
適用於 ViT / Swin Transformer 等同尺度層的殘差替換
論文: Attention Residuals (Kimi Team)
參考: https://github.com/MoonshotAI/Attention-Residuals
核心思想:
標準殘差: h_l = h_{l-1} + f_{l-1}(h_{l-1}) (固定均勻累加)
AttnRes: h_l = Σ α_{i→l} · v_i (softmax 加權選擇性聚合)
其中:
- Q = w_l (每層一個可學習的 d 維向量, 不依賴輸入)
- K = RMSNorm(v_i) (各層輸出經 RMSNorm)
- V = v_i (各層原始輸出)
- α_{i→l} = softmax(w_l^T · RMSNorm(v_i)) 沿深度維度歸一化
"""
import torch
import torch.nn as nn
from typing import List, Optional
# ============================================================
# 1. RMSNorm (論文中對 Key 使用的歸一化)
# ============================================================
class RMSNorm(nn.Module):
"""Root Mean Square Layer Normalization"""
def __init__(self, dim: int, eps: float = 1e-8):
super().__init__()
self.eps = eps
self.weight = nn.Parameter(torch.ones(dim))
def forward(self, x: torch.Tensor) -> torch.Tensor:
# x:...
GitHub Issue