SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

Keyphrases

• Low-rank 100%
• Memory Efficiency 100%
• Low-rank Structure 100%
• Fixed Support 100%
• Ranking Approach 100%
• Parameter Efficiency 100%
• Large Language Models 100%
• Learning Strategies 50%
• Sparsity 50%
• Llama 50%
• Matrix Factorization 50%
• Sparse Component 50%
• Memory Capacity 50%
• Memory Requirements 50%
• Low-dimensional Subspace 50%
• Full Rank 50%
• Sparse Matrices 50%
• Parameter Tuning 50%
• Memory Cost 50%
• Simple Strategy 50%
• Low-rank Matrix 50%
• Computational Power 50%
• Low-rank Component 50%
• Sparse Learning 50%
• Low-rank Factorization 50%
• Low-rank Adaptation 50%
• Low-rank Learning 50%

Computer Science

• Large Language Model 100%
• Sparsity 50%
• Memory Capacity 50%
• Matrix Factorization 50%
• Dimensional Subspace 50%
• Memory Requirement 50%
• Computational Power 50%
• Sparse Learning 50%