Switching to float for histogram building

[Shiki-H]

Hi, I'd like to get some clarifications about the decision to stick to double for GHistRow that is used in histogram building. @RAMitchell mentioned in this comment that switching to float will result in significant accuracy degradation. I'm just wondering how is performance measured in this case. More generally, if say we have ways to improve the runtime of model building that would lead to changes in model metrics, from the development team's perspective, what level of variation is acceptable?

[trivialfis]

Hi, XGBoost GPU used to have this option. I proposed and merged the removal of this option (a long time ago). The model wasn't converging when f32 was used for accumulation when the number of samples was large. This can be obvious for typical regression since one would see metrics like rmse increasing for the training dataset. But it's not apparent if the metric is in a log scale like logistic regression. In that case, the model can still be garbage even if the metric somewhat improves.

what level of variation is acceptable

As a result, it's not about variation. It's about converging or not, and whether a user can quickly tell he or she is running into trouble.

I don't have the experiment's results now. I think I was using the HIGGS dataset as a demonstration. The performance gain is negligible for small datasets, whereas for large datasets, it might not converge. The parameter's practical usefulness was small.

[trivialfis]

Feel free to close if there are no further questions.

[Shiki-H]

@trivialfis thanks, that was very helpful