SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2210.10325 · EMNLP · 2022

Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping

Xuezhe Ma, Chenghao Yang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Fine-tuning over large pretrained language models (PLMs) has established many stateof-the-art results. Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications. Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively fine-tuning layers in top-down manner is a promising solution. In this paper, we first point out that this method does not always work out due to different convergence speeds of different layers/modules. Inspired by this observation, we propose a simple componentwise gradient norm clipping method to adjust the convergence speed for different components. Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed and training stability. The codebase can be found at https://github.com/ yangalan123/FineTuningStability.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2210.10325")
get_code_for_paper("2210.10325")
have("2210.10325")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2210.10325.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2210.10325.svg)](https://syntology.ai/paper/2210.10325)

Connect an agent — have() is free.