SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2106.02205 · ACL · 2021

Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

Ji-Rong Wen, Wayne Zhao, Peiyu Liu, Zhong-Yi Lu, Ze-Feng Gao, Z Xie

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics. It can decompose an original matrix into central tensors (containing the core information) and auxiliary tensors (with only a small proportion of parameters). With the decomposed MPO structure, we propose a novel fine-tuning strategy by only updating the parameters from the auxiliary tensors, and design an optimization algorithm for MPO-based approximation over stacked network architectures. Our approach can be applied to the original or the compressed PLMs in a general way, which derives a lighter network and significantly reduces the parameters to be fine-tuned. Extensive experiments have demonstrated the effectiveness of the proposed approach in model compression, especially the reduction in finetuning parameters (91% reduction on average). The code to reproduce the results of this paper can be found at https://github.com/ RUCAIBox/MPOP.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2106.02205")
get_code_for_paper("2106.02205")
have("2106.02205")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2106.02205.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2106.02205.svg)](https://syntology.ai/paper/2106.02205)

Connect an agent — have() is free.