SPR-RAFT: Parameter-Efficient Regression-Aware Fine-Tuning for Biomedical LLM Regression

Abstract (source: OpenReview · © authors)

Biomedical regression tasks require predicting continuous targets from heterogeneous and unstructured evidence. While Large Language Models (LLMs) provide a robust interface for reasoning over mixed modalities, they are inherently limited by their discrete tokenization and cross-entropy objectives, which lack awareness of numerical proximity. To bridge this gap, we present \textbf{SPR-RAFT}, a parameter-efficient and regression-aware framework that adapts frozen LLMs for high-precision regression. SPR-RAFT introduces a dual-module architecture: a learnable soft prompt that conditions the LLM to route numerical reasoning into a specific latent state, and a lightweight regression head anchored on a dedicated readout token for numerical reasoning consolidation. Crucially, we align these two modalities via a hybrid objective that combines distribution-based text generation with representation-based robust regression. This ensures the model remains both semantically coherent and numerically calibrated. With only about 1.6M trainable parameters ($\sim$0.04\% of a 4B-parameter backbone), SPR-RAFT consistently outperforms prompting strategies, standard fine-tuning, and non-LLM baselines across diverse biomedical benchmarks, including clinical trial duration, biological age estimation, and molecular property prediction.

Keywords