Since Cox’s time varying proportional hazard model doesn’t appear to be implemented in dask_ml, and the lifelines package doesn’t appear to offer integration with dask, is there any way I can leverage dask to fit lifelines’ CoxTimeVaryingFitter() to a large training set?
Hi @MarkPundurs, welcome to this forum.
If you want to fit a model on a dataset larger than memory using Dask, as stated in
https://ml.dask.org/incremental.html
the model must at least implement the partial_fit API from Sickit-Learn.
By looking at lifelines package documentation, it seems it does not support it.
I guess there is no magic wand here, either the algorithm is compatible with this and someone needs to implement this, either this is not possible to fit this model on a dataset larger then memory using Dask.
Hope that helps.