"IntigercastingNaNError: Cannot convert non-finite value (NA or inf) to integer"

Dask DataFrame
1

Description:
I am working on a migration project where I am using Dask on top of FastAPI to pull data from Snowflake and store it in PostgreSQL. I receive the data as a list of dictionaries. I have encountered an issue when handling null values and trying to use the compute() function in Dask. I consistently receive the following error: ‘IntegerCastingNaNError: Cannot convert non-finite value (NA or INF) to integer.’

I have attempted the following approaches to resolve the issue, but none have been successful:

  • Defining each column with a specific data type.
  • Replacing NaN and INF values.
  • Applying a lambda function using map_partitions to remove INF or -INF values.

I would appreciate any guidance or suggestions on how to handle this issue and successfully compute the Dask DataFrame without encountering the IntegerCastingNaNError.
Thank you in advance for your help.

2

Hi @uannabi, welcome to Dask Discourse forum!

It would be really helpful if you could post a minimal reproducer, with an example of dictionary in input that throws the error.

Have you tried the solution proposed here, using meta argument when converting to DataFrame?

3

I really appreciate your replay. This method can be applied for few columns. I’ve a quite number of columns, this data are coming from snowflake as a list of dictionary.

4

In the same github threads, there are proposed solutions to do it automatically after retrieving the list of columns, see IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer on non integer column. · Issue #8386 · dask/dask · GitHub.

5

That did not work in the given case. Is it possible to post a sample code snippet reffering to database table?