Jian Yang, Xun Zhou, Zhijian Zhuo, Jinwen Ma, Ya Wang, Yutao Zeng, Xiaoqing Li, Sijun Zhang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, many challenges remain in training deep transformer networks, especially regarding the position of the layer normalization. While Pre-Norm structures facilitate more stable training owing to their stronger identity path, they often lead to suboptimal performance compared to Post-Norm. In this paper, we propose HybridNorm, a simple yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. We provide both theoretical insights and empirical evidence to demonstrate that HybridNorm improves the gradient flow and the model robustness. Extensive experiments on large-scale transformer models, including both dense and sparse variants, show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches across multiple benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2503.04598")
get_code_for_paper("2503.04598")
have("2503.04598")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2503.04598.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2503.04598)
Connect an agent — have() is free.