Election-Based Task Pruning in Mixture-of-Experts for Scalable Multi-Task Learning

Scientific analyses often involve multiple tasks that are closely related to each other. These tasks inherently share dependencies that can be effectively modeled through multi-task learning (MTL), which enables shared representation learning and cross-task generalization. However, applying MTL to complex scientific datasets with numerous, weakly related tasks remains challenging due to the computational burden of exploring extensive task combinations and the lack of automated task selection strategies. To address these challenges, we propose Self Expert Exploration for Mixture of Experts models (SEEM), a novel MTL pruning framework that adaptively identifies and retains only the most relevant tasks during training. SEEM introduces an importance-based pruning algorithm and a two-tier grouping mechanism distinguishing primary and secondary tasks. It quantifies inter-task relationships via gating patterns, graph-based centrality, and consensus voting to drive pruning decisions. By dynamically pruning tasks, SEEM produces a compact, computationally efficient model while preserving shared learning performance and enhancing computing efficiency. We evaluate SEEM using advanced Mixture of Experts architectures across multiple scientific and real-world datasets, spanning both regression and classification tasks. Our results demonstrate that SEEM achieves superior scalability and reduced computational cost, while improving interpretability by revealing key inter-task relationships and mitigating negative transfer from weakly related tasks.