Generalization-Aware Evaluation of HLS QoR Prediction: In-Distribution Performance versus Cross-Algorithm Generalization Using ForgeHLS

Abstract

High-Level Synthesis (HLS) enables efficient FPGA-based accelerator design, but quality-of-results (QoR) exploration remains expensive because accurate latency and resource estimates typically require full synthesis flows. This has motivated the use of machine
learning (ML) for early QoR prediction. However, it remains unclear whether strong performance under conventional random train/test splits truly reflects generalization to unseen algorithms. In this work, we study this question using the ForgeHLS design-level dataset. We build a reproducible multitarget QoR prediction pipeline to estimate six metrics: BRAM, LUT, DSP, FF, best-case latency, and worst-case latency. To isolate the effect of the evaluation protocol, we adopt a simple and interpretable Random Forest baseline and evaluate it under two settings: a standard random split and a stricter group split based on algorithm identity. In addition, a boosting-based baseline (XGBoost) is evaluated to verify that the observed behavior is not model-specific. We further repeat the group-based evaluation across multiple random seeds. The results show that the baseline performs strongly under random splitting, especially for BRAM and latency-related targets, but that its performance degrades sharply under group-based evaluation. Repeated group-split experiments confirm that this degradation is systematic rather than incidental. Feature importance analysis further indicates that the strongest predictive signals are closely tied to algorithm identity and source-related information. Overall, the study shows that random-split evaluation can substantially overestimate the true capability of HLS QoR predictors and underscores the importance of generalization-aware evaluation protocols for future ML-based HLS QoR prediction research.