[Survey] Recent approaches about Efficient ML

This is a collection of recent approaches and papers about Efficient ML including Parameter Efficient Fine Tuning(PEFT), qunatization, pruning and other topics.

PEFT

• MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning(arxiv)
• LayerNorm: A key component in parameter-efficient fine-tuning(arxiv)
• ReFT: Representation Finetuning for Language Models(arxiv)
• LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning(arxiv)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection(arxiv)
• LORAPRUNE: PRUNING MEETS LOW-RANK PARAMETER-EFFICIENT FINE-TUNING(openreview)
• LoRA+: Efficient Low Rank Adaptation of Large Models(arxiv)

Quantization

• Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models(arxiv)
• QLoRA: Efficient Finetuning of Quantized LLMs(arixv)

Pruning

• Random Search as a Baseline for Sparse Neural Network Architecture Search(arxiv)
• The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models(arxiv)