I am working on a migration project where I am using Dask on top of FastAPI to pull data from Snowflake and store it in PostgreSQL. I receive the data as a list of dictionaries. I have encountered an issue when handling null values and trying to use the
compute() function in Dask. I consistently receive the following error: ‘IntegerCastingNaNError: Cannot convert non-finite value (NA or INF) to integer.’
I have attempted the following approaches to resolve the issue, but none have been successful:
- Defining each column with a specific data type.
- Replacing NaN and INF values.
- Applying a lambda function using
map_partitionsto remove INF or -INF values.
I would appreciate any guidance or suggestions on how to handle this issue and successfully compute the Dask DataFrame without encountering the IntegerCastingNaNError.
Thank you in advance for your help.