Dask.dataframe.from_pandas gives error {FutureCancelledError}FutureCancelledError()

Dask DataFrame
1

I have:

Platform win32
Python ‘3.11.10 | packaged by Anaconda, Inc. | (main, Oct 3 2024, 07:22:26) [MSC v.1929 64 bit (AMD64)]’
DASK version 2024.8.2
Pandas version ‘2.2.2’

pd_df is Pandas dataframe:

                'col1': dtype('int64'),
                'col2': dtype('O'),
                'col3': dtype('int64'),
                'col4': dtype('int64'),
                'col5': dtype('O'),
                'col6': dtype('O'),
                'col7': dtype('O'),
                'col8': dtype('float64'),
                'col9': dtype('float64'),
                'col10': dtype('float64'),
                'col11': dtype('float64'),
                'col12': dtype('float64'),
                'col13': dtype('O'),
                'col14': dtype('O'),
                'col15': dtype('O'),
                'col16': dtype('O'),
                'col17': dtype('O'),
                'col18': dtype('O')

The
import dask.dataframe as dd
dd.from_pandas(pd_df, npartitions =16 ).compute()

getting error:
{FutureCancelledError}FutureCancelledError()

dask.dataframe.from_pandas

No issue for

Python ‘3.10.11 | packaged by Anaconda, Inc. | (main, May 16 2023, 00:55:32) [MSC v.1916 64 bit (AMD64)]’
DASK 2024.3.1
Pandas 2.2.0

2

Hi @slepeturin,

Do you have a reproducer of this problem, I mean a real or generated dataset that triggers it?

What is the complete stack trace of the Exception?

3

Hi @@guillaumeeb,
Thank you for your response!
I could not provide a complete stack as I upgraded the environment to conda-forge: conda update -c conda-forge --all.
I’m running fine for now with package list below:
name: myenv_311
channels:

  • defaults
  • conda-forge
    dependencies:
  • _openmp_mutex=4.5=2_gnu
  • arrow-cpp=13.0.0=h57928b3_49_cpu
  • asttokens=2.4.1=pyhd8ed1ab_0
  • aws-c-auth=0.7.22=h8c86ca4_10
  • aws-c-cal=0.7.1=hea5f451_1
  • aws-c-common=0.9.23=h2466b09_0
  • aws-c-compression=0.2.18=hea5f451_7
  • aws-c-event-stream=0.4.2=h4b8288a_15
  • aws-c-http=0.8.2=h269d64e_6
  • aws-c-io=0.14.10=hfca834b_1
  • aws-c-mqtt=0.10.4=h519d897_8
  • aws-c-s3=0.6.0=hb746b11_2
  • aws-c-sdkutils=0.1.16=hea5f451_3
  • aws-checksums=0.1.18=hea5f451_7
  • aws-crt-cpp=0.27.3=h8c89294_2
  • aws-sdk-cpp=1.11.329=he0aa860_9
  • binutils_impl_win-64=2.43=h095e170_2
  • blas=1.0=mkl
  • bokeh=3.6.0=pyhd8ed1ab_0
  • boost-cpp=1.85.0=ha5ead02_4
  • boto3=1.35.50=pyhd8ed1ab_0
  • botocore=1.35.50=pyge310_1234567_0
  • bottleneck=1.4.2=py311h0a17f05_0
  • brotli-python=1.1.0=py311hda3d55a_2
  • bzip2=1.0.8=h2466b09_7
  • c-ares=1.34.2=h2466b09_0
  • ca-certificates=2024.9.24=haa95532_0
  • cffi=1.17.1=py311he736701_0
  • clang_variant=1.0=default
  • clangdev=5.0.0=h5aa907a_1012
  • click=8.1.7=win_pyh7428d3b_0
  • cloudpickle=3.1.0=pyhd8ed1ab_1
  • cmake=3.30.5=h400e5d1_0
  • colorama=0.4.6=pyhd8ed1ab_0
  • contourpy=1.3.0=py311h3257749_2
  • cx_oracle=8.3.0=py311h984d3dc_4
  • cython=3.0.11=py311hdd1c356_3
  • cytoolz=1.0.0=py311he736701_1
  • dask=2024.8.1=pyhd8ed1ab_0
  • dask-core=2024.8.1=pyhd8ed1ab_0
  • dask-expr=1.1.11=pyhd8ed1ab_0
  • distributed=2024.8.1=pyhd8ed1ab_0
  • executing=2.1.0=pyhd8ed1ab_0
  • freetype=2.12.1=hdaf720e_2
  • fsspec=2024.10.0=pyhff2d567_0
  • gcc=14.2.0=ha8167b9_1
  • gcc_impl_win-64=14.2.0=h4b1235c_1
  • gflags=2.2.2=he0c23c2_1005
  • git=2.47.0=h57928b3_0
  • glog=0.7.1=h3ff59bf_0
  • greenlet=3.1.1=py311hda3d55a_0
  • gxx=14.2.0=ha8167b9_1
  • gxx_impl_win-64=14.2.0=hb404c5b_1
  • h2=4.1.0=pyhd8ed1ab_0
  • halo=0.0.31=pyhd8ed1ab_0
  • heapdict=1.0.1=pyhd8ed1ab_1
  • hpack=4.0.0=pyh9f0ad1d_0
  • hyperframe=6.0.1=pyhd8ed1ab_0
  • importlib-metadata=8.5.0=pyha770c72_0
  • importlib_metadata=8.5.0=hd8ed1ab_0
  • intel-openmp=2024.2.1=h57928b3_1083
  • jinja2=3.1.4=pyhd8ed1ab_0
  • jmespath=1.0.1=pyhd8ed1ab_0
  • jpeg=9e=hcfcfb64_3
  • krb5=1.21.3=hdf4eb48_0
  • lcms2=2.15=ha5c8aab_0
  • ld_impl_win-64=2.43=hae1bf67_2
  • lerc=4.0.0=h63175ca_0
  • libabseil=20240116.2=cxx17_he0c23c2_1
  • libarrow=13.0.0=h93f6c72_49_cpu
  • libboost=1.85.0=h444863b_4
  • libboost-devel=1.85.0=h91493d7_4
  • libboost-headers=1.85.0=h57928b3_4
  • libbrotlicommon=1.1.0=h2466b09_2
  • libbrotlidec=1.1.0=h2466b09_2
  • libbrotlienc=1.1.0=h2466b09_2
  • libcrc32c=1.1.2=h0e60522_0
  • libcurl=8.10.1=h1ee3ff0_0
  • libdeflate=1.22=h2466b09_0
  • libevent=2.1.12=h3671451_1
  • libexpat=2.6.3=he0c23c2_0
  • libffi=3.4.4=hd77b12b_1
  • libflang=5.0.0=h6538335_20180525
  • libgcc=14.2.0=h1383e82_1
  • libgcc-devel_win-64=14.2.0=h7c23ffb_101
  • libgomp=14.2.0=h1383e82_1
  • libgoogle-cloud=2.26.0=h5e7cea3_0
  • libgoogle-cloud-storage=2.26.0=he5eb982_0
  • libgrpc=1.62.2=h5273850_0
  • libiconv=1.17=hcfcfb64_2
  • libopenblas=0.3.28=pthreads_hf0a32cb_0
  • libpng=1.6.44=h3ca93ac_0
  • libprotobuf=4.25.3=h47a098d_1
  • libre2-11=2023.09.01=hf8d8778_2
  • libsqlite=3.47.0=h2466b09_1
  • libssh2=1.11.0=h7dfc565_0
  • libstdcxx=14.2.0=h904f734_1
  • libstdcxx-devel_win-64=14.2.0=h7c23ffb_101
  • libthrift=0.19.0=ha2b3283_1
  • libtiff=4.5.0=h8a3f274_0
  • libutf8proc=2.8.0=h82a8f57_0
  • libuv=1.49.2=h2466b09_0
  • libwebp-base=1.4.0=hcfcfb64_0
  • libwinpthread=12.0.0.r4.gg4f2fc60ca=h57928b3_8
  • libxcb=1.13=0
  • libzlib=1.3.1=h2466b09_2
  • llvm-meta=13.0.1=hd8ed1ab_0
  • locket=1.0.0=pyhd8ed1ab_0
  • log_symbols=0.0.14=pyh8c360ce_0
  • lz4=4.3.3=py311h8b5e962_1
  • lz4-c=1.9.4=hcfcfb64_0
  • m2w64-sysroot_win-64=12.0.0.r4.gg4f2fc60ca=h7428d3b_8
  • markupsafe=3.0.2=py311h5082efb_0
  • mingw-w64-ucrt-x86_64-crt-git=12.0.0.r4.gg4f2fc60ca=h7428d3b_8
  • mingw-w64-ucrt-x86_64-headers-git=12.0.0.r4.gg4f2fc60ca=h7428d3b_8
  • mingw-w64-ucrt-x86_64-windows-default-manifest=6.4=he206cdd_7
  • mingw-w64-ucrt-x86_64-winpthreads-git=12.0.0.r4.gg4f2fc60ca=h7428d3b_8
  • mkl=2023.2.0=h6a75c08_49573
  • mkl-service=2.4.1=py311h130b7e8_1
  • mkl_fft=1.3.10=py311h1d15ed5_1
  • mkl_random=1.2.8=py311h97a4ddd_0
  • msgpack-python=1.1.0=py311h3257749_0
  • numexpr=2.8.7=py311h1fcbade_0
  • numpy=1.26.4=py311hdab7c0b_0
  • numpy-base=1.26.4=py311hd01c5d8_0
  • openjpeg=2.5.2=hae555c5_0
  • openmp=5.0.0=vc14_0
  • openssl=3.3.2=h2466b09_0
  • orc=2.0.1=h7e885a9_1
  • packaging=24.1=pyhd8ed1ab_0
  • pandas=2.2.3=py311hcf9f919_1
  • parquet-tools=0.2.16=pyhd8ed1ab_0
  • partd=1.4.2=pyhd8ed1ab_0
  • pillow=9.4.0=py311h76d9071_1
  • pip=24.3.1=pyh8b19718_0
  • psutil=6.1.0=py311he736701_0
  • pyarrow=13.0.0=py311h4452abf_49_cpu
  • pyarrow-hotfix=0.6=pyhd8ed1ab_0
  • pycparser=2.22=pyhd8ed1ab_0
  • pygments=2.18.0=pyhd8ed1ab_0
  • pyodbc=5.2.0=py311hda3d55a_0
  • pysocks=1.7.1=pyh0701188_6
  • pysqlite3=0.5.4=py313hb2a9d6d_0
  • python=3.11.10=hce54a09_3_cpython
  • python-dateutil=2.9.0post0=py311haa95532_2
  • python-devtools=0.12.2=pyhd8ed1ab_0
  • python-lmdb=1.5.1=py311hb5d55ac_1
  • python-tzdata=2024.2=pyhd8ed1ab_0
  • python_abi=3.11=5_cp311
  • pytz=2024.1=pyhd8ed1ab_0
  • pyyaml=6.0.2=py311he736701_1
  • re2=2023.09.01=hd3b24a8_2
  • s3transfer=0.10.3=pyhd8ed1ab_0
  • setuptools=75.1.0=pyhd8ed1ab_0
  • six=1.16.0=pyh6c4a22f_0
  • snappy=1.2.1=h23299a8_0
  • sortedcontainers=2.4.0=pyhd8ed1ab_0
  • spinners=0.0.24=pyh9f0ad1d_0
  • sqlalchemy=2.0.36=py311he736701_0
  • sqlite=3.47.0=h2466b09_1
  • tabulate=0.9.0=pyhd8ed1ab_1
  • tbb=2021.8.0=h59b6b97_0
  • tblib=3.0.0=pyhd8ed1ab_0
  • termcolor=2.5.0=pyhd8ed1ab_0
  • thrift=0.20.0=py311hda3d55a_1
  • tk=8.6.13=h5226925_1
  • toolz=1.0.0=pyhd8ed1ab_0
  • tornado=6.4.1=py311he736701_1
  • typing-extensions=4.12.2=hd8ed1ab_0
  • typing_extensions=4.12.2=pyha770c72_0
  • tzdata=2024b=hc8b5060_0
  • ucrt=10.0.22621.0=h57928b3_1
  • urllib3=2.2.3=pyhd8ed1ab_0
  • utf8proc=2.6.1=h2bbff1b_1
  • vc=14.40=h2eaa2aa_1
  • vc14_runtime=14.40.33810=hcc2c482_22
  • vs2015_runtime=14.40.33810=h3bf8584_22
  • wheel=0.44.0=pyhd8ed1ab_0
  • win_inet_pton=1.1.0=pyh7428d3b_7
  • xarray=2024.10.0=pyhd8ed1ab_0
  • xorg-libxau=1.0.11=h0e40799_1
  • xorg-libxdmcp=1.1.5=h0e40799_0
  • xyzservices=2024.9.0=pyhd8ed1ab_0
  • xz=5.4.6=h8cc25b3_1
  • yaml=0.2.5=h8ffe710_2
  • zict=3.0.0=pyhd8ed1ab_0
  • zipp=3.20.2=pyhd8ed1ab_0
  • zlib=1.3.1=h2466b09_2
  • zstandard=0.23.0=py311h53056dc_1
  • zstd=1.5.6=h0ea2cb4_0
4

OK, so maybe there was a bug with your workflow in a precise Dask version.