Attention-Aware Routing: Coupling Routing and Attention in MoEs

事件

arXiv:2609.20974v1 Announce Type: new Abstract: In Mixture-of-Experts language models, the router typically selects and weights experts based on the token's hidden state, utilizing limited contextual information. We propose Attention-Aware Routing (AAR), which augments the router with temporal and s

来源

本条目由采集管线自动抓取并发布，完整内容见下方来源链接。

*采集源：arXiv cs.AI*