JET Expansions of Residual Computation

Overview

This project introduces a systematic framework called JET Expansion, which uses jet operators to expand residual computation into interpretable paths. Unlike other methods, JET Expansion requires no data or additional training, offering a direct way to analyze the computational behavior of models like transformers, where the residual computation plays a vital role.

• Provides insights into the inner workings of large language models with jet lens.
• Reformat language models as symbolic databases with jet n-grams for model diffing
• Enables data-free analysis of component roles, pretraining dynamic, and finetuning effects with applications like toxicity analysis.

Visualizations

Technical Details

The method relies on recursively applying jet expansions, which generalize Taylor series, providing a way to disentangle different computational paths in deep learning models like transformers. It does not require additional data, training, or sampling. Our paper offers more details about the method.

Use Cases

Below are examples of how JET Expansion can be used to gain insights into popular models:

• Model Diff Diff language models using jet n-grams.
• Trace Pretraining Dynamic Trace the evolution during pretraining.
• Data-Free Toxicity Indexing Measure the toxicity embedded in models before and after RLHF fine-tuning.
• Jet Lens for Understanding Inner Workings

BibTeX

    @misc{chen2024jetexpansionsresidualcomputation,
        title={Jet Expansions of Residual Computation}, 
        author={Yihong Chen and Xiangxiang Xu and Yao Lu and Pontus Stenetorp and Luca Franceschi},
        year={2024},
        eprint={2410.06024},
        archivePrefix={arXiv},
        primaryClass={cs.LG},
        url={https://arxiv.org/abs/2410.06024}
    }
        

Contact

For more information or inquiries, please reach out via email.