@mvashishtha Looking at the issue you’ve linked to, some thoughts:
- the failures seem to have started somewhere between 9 days ago to 4 days ago (based on CI runs)
- since the Dask version is pinned, we think this might be related to some other package that changed between ci passing and failing
Here’s a diff between this(passing) and this(failing) – red is failing version, green is passing:
Full diff:
diff --git a/modin-ubuntu-dask-engine-fail.txt b/modin-ubuntu-dask-engine-pass.txt
index 154ee88..2cf16fa 100644
--- a/modin-ubuntu-dask-engine-fail.txt
+++ b/modin-ubuntu-dask-engine-pass.txt
@@ -12,11 +12,11 @@ aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge
alabaster 0.7.12 pypi_0 pypi
argon2-cffi 21.3.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
-arrow-cpp 8.0.1 py38ha7276ea_0_cpu conda-forge
+arrow-cpp 8.0.0 py38ha7276ea_4_cpu conda-forge
asttokens 2.0.5 pyhd8ed1ab_0 conda-forge
asv 0.5.dev1889+ef016e23 pypi_0 pypi
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
-attrs 22.1.0 pyh71513ae_1 conda-forge
+attrs 21.4.0 pyhd8ed1ab_0 conda-forge
aws-c-cal 0.5.11 h95a6274_0 conda-forge
aws-c-common 0.6.2 h7f98852_0 conda-forge
aws-c-event-stream 0.2.7 h3541f99_13 conda-forge
@@ -69,7 +69,7 @@ et_xmlfile 1.0.1 py_1001 conda-forge
execnet 1.9.0 pyhd8ed1ab_0 conda-forge
executing 0.9.1 pyhd8ed1ab_0 conda-forge
faker 13.15.1 pypi_0 pypi
-fastavro 1.5.4 py38h0a891b7_0 conda-forge
+fastavro 1.5.3 py38h0a891b7_0 conda-forge
fastjsonschema 2.16.1 pypi_0 pypi
feather-format 0.4.1 pyh9f0ad1d_0 conda-forge
filelock 3.7.1 pypi_0 pypi
@@ -77,7 +77,7 @@ flake8 4.0.1 pypi_0 pypi
freetds 1.1.15 h94af77a_1 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
frozenlist 1.3.0 py38h0a891b7_1 conda-forge
-fsspec 2022.7.1 pyhd8ed1ab_0 conda-forge
+fsspec 2022.5.0 pyhd8ed1ab_0 conda-forge
fuzzydata 0.0.6 pypi_0 pypi
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
@@ -100,7 +100,7 @@ grpc-cpp 1.45.2 h3b8df00_4 conda-forge
grpcio 1.43.0 pypi_0 pypi
hdf5 1.12.1 nompi_h2386368_104 conda-forge
heapdict 1.0.1 py_0 conda-forge
-icu 67.1 he1b5a44_0 conda-forge
+icu 58.2 hf484d3e_1000 conda-forge
idna 3.3 pyhd8ed1ab_0 conda-forge
imagesize 1.4.1 pypi_0 pypi
importlib-metadata 4.12.0 pypi_0 pypi
@@ -115,7 +115,7 @@ jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h166bdaf_2 conda-forge
-jsonschema 4.9.0 pypi_0 pypi
+jsonschema 4.7.2 pypi_0 pypi
jupyter 1.0.0 pypi_0 pypi
jupyter-client 7.3.4 pypi_0 pypi
jupyter-console 6.4.4 pypi_0 pypi
@@ -127,7 +127,7 @@ kiwisolver 1.4.4 py38h43d8883_0 conda-forge
krb5 1.19.3 h3790be6_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
-lerc 4.0.0 h27087fc_0 conda-forge
+lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 15_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_7 conda-forge
libbrotlidec 1.0.9 h166bdaf_7 conda-forge
@@ -158,29 +158,29 @@ libsodium 1.0.18 h36c2ea0_1 conda-forge
libssh2 1.10.0 ha56f1ee_2 conda-forge
libstdcxx-ng 12.1.0 ha89aaad_16 conda-forge
libthrift 0.16.0 h519c5ea_1 conda-forge
-libtiff 4.4.0 h0d92c0b_2 conda-forge
+libtiff 4.4.0 hc85c160_1 conda-forge
libutf8proc 2.7.0 h7f98852_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp 1.2.3 h522a892_1 conda-forge
libwebp-base 1.2.3 h166bdaf_2 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
-libxml2 2.9.10 h68273f3_2 conda-forge
-libxslt 1.1.33 hf705e74_1 conda-forge
+libxml2 2.9.14 h74e7548_0
+libxslt 1.1.35 h4e12654_0
libzlib 1.2.12 h166bdaf_2 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
-lxml 4.8.0 py38h0a891b7_3 conda-forge
+lxml 4.9.1 py38h0a891b7_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
lzo 2.10 h516909a_1000 conda-forge
markupsafe 2.1.1 py38h0a891b7_1 conda-forge
matplotlib 3.2.2 1 conda-forge
-matplotlib-base 3.2.2 py38h5d868c9_1 conda-forge
+matplotlib-base 3.2.2 py38hef1b27d_0
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
mccabe 0.6.1 pypi_0 pypi
mistune 0.8.4 pypi_0 pypi
modin-spreadsheet 0.1.2 pypi_0 pypi
msgpack-python 1.0.4 py38h43d8883_0 conda-forge
multidict 6.0.2 py38h0a891b7_1 conda-forge
-mypy 0.971 py38h0a891b7_0 conda-forge
+mypy 0.961 py38h0a891b7_0 conda-forge
mypy_extensions 0.4.3 py38h578d9bd_5 conda-forge
nbclient 0.6.6 pypi_0 pypi
nbconvert 6.5.0 pypi_0 pypi
@@ -204,7 +204,7 @@ orc 1.7.5 h6c59b99_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.4.3 py38h47df419_0 conda-forge
pandas-gbq 0.17.6 pyh6c4a22f_0 conda-forge
-pandas-stubs 1.4.3.220801 pyhd8ed1ab_0 conda-forge
+pandas-stubs 1.4.3.220724 pyhd8ed1ab_0 conda-forge
pandocfilters 1.5.0 pypi_0 pypi
paramiko 2.11.0 pyhd8ed1ab_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
@@ -214,8 +214,7 @@ pathspec 0.9.0 pypi_0 pypi
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.2.0 py38h0ee0e06_0 conda-forge
-pip 22.2.1 pyhd8ed1ab_0 conda-forge
-pkgutil-resolve-name 1.3.10 pypi_0 pypi
+pip 22.2 pyhd8ed1ab_0 conda-forge
platformdirs 2.5.2 pypi_0 pypi
pluggy 1.0.0 py38h578d9bd_3 conda-forge
plumbum 1.7.2 pyhd8ed1ab_0 conda-forge
@@ -231,7 +230,7 @@ pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
py 1.11.0 pyh6c4a22f_0 conda-forge
py-cpuinfo 8.0.0 pyhd8ed1ab_0 conda-forge
py-spy 0.3.12 pypi_0 pypi
-pyarrow 8.0.1 py38h9f6a473_0_cpu conda-forge
+pyarrow 8.0.0 py38h9f6a473_4_cpu conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycodestyle 2.8.0 pypi_0 pypi
@@ -249,7 +248,7 @@ pyrsistent 0.18.1 pypi_0 pypi
pysocks 1.7.1 py38h578d9bd_5 conda-forge
pytables 3.7.0 py38hdb04529_0 conda-forge
pytest 7.1.2 py38h578d9bd_0 conda-forge
-pytest-benchmark 3.4.1 pyhd8ed1ab_1 conda-forge
+pytest-benchmark 3.4.1 pyhd8ed1ab_0 conda-forge
pytest-cov 2.11.0 pyh44b312d_0 conda-forge
pytest-forked 1.4.0 pyhd8ed1ab_0 conda-forge
pytest-xdist 2.5.0 pyhd8ed1ab_0 conda-forge
@@ -271,10 +270,10 @@ requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rpyc 4.1.5 pyh9f0ad1d_1 conda-forge
rsa 4.9 pyhd8ed1ab_0 conda-forge
s2n 1.0.10 h9b69904_0 conda-forge
-s3fs 2022.7.1 pyhd8ed1ab_0 conda-forge
+s3fs 2022.5.0 pyhd8ed1ab_0 conda-forge
s3transfer 0.5.2 pyhd8ed1ab_0 conda-forge
scikit-learn 1.1.1 py38hf80bbf7_0 conda-forge
-scipy 1.9.0 py38hea3f02b_0 conda-forge
+scipy 1.8.1 py38hea3f02b_2 conda-forge
send2trash 1.8.0 pypi_0 pypi
setuptools 59.8.0 py38h578d9bd_1 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
@@ -309,7 +308,7 @@ typing_extensions 4.3.0 pyha770c72_0 conda-forge
typing_inspect 0.7.1 pyh6c4a22f_0 conda-forge
unixodbc 2.3.10 h583eb01_0 conda-forge
urllib3 1.26.11 pyhd8ed1ab_0 conda-forge
-virtualenv 20.16.2 pypi_0 pypi
+virtualenv 20.16.1 pypi_0 pypi
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 pypi_0 pypi
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
Dask’s test suite was also stalling around the same time, see: Remove `werkzeug` pin in CI · Issue #9323 · dask/dask · GitHub (and relevant PR)
With this context, maybe the issue is in the s3fs
version:
-s3fs 2022.7.1 pyhd8ed1ab_0 conda-forge
+s3fs 2022.5.0 pyhd8ed1ab_0 conda-forge
You can try pinning s3fs
or the same package as Dask(werkzeug
) and see if that helps?