Karthikeyan Ramamurthy, Vijil Chenthamarakshan, Dennis Wei, Amit Dhurandhar, Rahul Nair
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformers are arguably the preferred architecture for language generation. In this paper, inspired by continued fractions, we introduce a new function class for generative modeling. The architecture family implementing this function class is named CoFrGeNets -Continued Fraction Generative Networks. We design novel architectural components based on this function class that can replace Multihead Attention and Feed-Forward Networks in Transformer blocks while requiring much fewer parameters. We derive custom gradient formulations to optimize the proposed components more accurately and efficiently than using standard PyTorch-based gradients. Our components are a plug-in replacement requiring little change in training or inference procedures that have already been put in place for Transformer-based models thus making our approach easy to incorporate in large industrial workflows. We experiment on two very different transformer architectures GPT2-xl (1.5B) and Llama3 (3.2B), where the former we pre-train on OpenWebText and GneissWeb, while the latter we pre-train on the docling data mix which consists of nine different datasets. Results show that the performance on downstream classification, Q& A, reasoning and text understanding tasks of our models is competitive and sometimes even superior to the original models with 2 3 to 1 2 the parameters and shorter pre-training time. We believe that future implementations customized to hardware will further bring out the true potential of our architectures. Preprint. Under review.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2601.21766")
get_code_for_paper("2601.21766")
have("2601.21766")
Connect an agent — have() is free.