# To\_sql() query does not work for large files out-of-memory on dask cluster inside docker
**URL:** https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378
**Category:** Dask DataFrame
**Created:** [February 18, 2022, 4:01pm UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378 "2022-02-18T16:01:18Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@SOUMYASHUKLA](https://dask.discourse.group/u/SOUMYASHUKLA)
#### Post date: [February 18, 2022, 4:01pm UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/1 "2022-02-18T16:01:18Z")
I am trying to create a sql table from a large (4GiB) csv file by using following dask command, on a dask cluster with 2 workers (1GiB memeory each), to simulate out-of-memory computation.
```auto
ddata = dd.read_csv(file_path, blocksize=raw_blocksize).set_index('Unnamed: 0')
ddata.to_sql(table_name, PostgresDBConnection().postgres_url, parallel=True, if_exists='replace')
```
This code works fine for files upto 2GiB, but for larger files below error is seen contineously on console logs and after writing 50% of data into sql, the process hangs.
```auto
dask_worker_1 | distributed.worker - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker.html#memtrim for more information. -- Unmanaged memory: 728.89 MiB -- Worker memory limit: 1.00 GiB
dask_worker_2 | distributed.worker - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker.html#memtrim for more information. -- Unmanaged memory: 783.55 MiB -- Worker memory limit: 1.00 GiB
```
---
### Author:  [@SOUMYASHUKLA](https://dask.discourse.group/u/SOUMYASHUKLA)
#### Post date: [February 18, 2022, 4:37pm UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/2 "2022-02-18T16:37:12Z")
This is the status on dask dashboard when the sql writing process is hung with continuous WARNING traceback as mentioned above.
Ideally, I was expecting the grey horizontal bars (indicating memory spill) while the process still executes. Am I missing out something?
---
### Author:  [@SOUMYASHUKLA](https://dask.discourse.group/u/SOUMYASHUKLA)
#### Post date: [February 18, 2022, 4:54pm UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/3 "2022-02-18T16:54:18Z")
I was hoping this [github bug](https://github.com/dask/dask/issues/2866) would help me with the fix.
However, no luck
---
### Author:  [@scharlottej13](https://dask.discourse.group/u/scharlottej13)
#### Post date: [February 19, 2022, 12:07am UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/4 "2022-02-19T00:07:14Z")
Hi @SOUMYASHUKLA, thanks for this question! You might find [this post](https://coiled.io/blog/tackling-unmanaged-memory-with-dask/) on tackling unmanaged memory helpful. You’re already doing the first step mentioned (tracking with the dashboard), but there are additional suggestions such as garbage collection and memory trimming that may help.
Do you have the same issue when saving the file to disk, or is this specific to loading to SQL, as the title of your post suggests?
---
### Author:  [@SOUMYASHUKLA](https://dask.discourse.group/u/SOUMYASHUKLA)
#### Post date: [February 19, 2022, 1:16am UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/5 "2022-02-19T01:16:24Z")
Hello @scharlottej13. Thank you for your response.
I have already tried the memory trim methods as mentioned in this post, however the memory problems still persists. This issue is specific to loading data into sql.
---
### Author:  [@pavithraes](https://dask.discourse.group/u/pavithraes)
#### Post date: [February 21, 2022, 1:51pm UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/6 "2022-02-21T13:51:40Z")
@SOUMYASHUKLA I’m again wondering if increasing the number of partitions in `ddata` will help?
---
### Author:  [@SOUMYASHUKLA](https://dask.discourse.group/u/SOUMYASHUKLA)
#### Post date: [February 23, 2022, 1:47am UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/7 "2022-02-23T01:47:23Z")
@pavithraes Thank you for your response. I did try reducing the blocksize to increase the number of partitions. However, to get things working, I ended up looping through each partition to save the data into postgres. 🙂
---
### Author:  [@pavithraes](https://dask.discourse.group/u/pavithraes)
#### Post date: [February 24, 2022, 6:33am UTC](https://dask.discourse.group/t/to-sql-query-does-not-work-for-large-files-out-of-memory-on-dask-cluster-inside-docker/378/8 "2022-02-24T06:33:41Z")
@SOUMYASHUKLA Thanks for sharing, I’m happy you found a work-around! I still feel Dask’s `to_sql()` should be able to handle this directly, so I’ll continue thinking about this.
One idea that comes to mind, `to_sql` triggers computation if you don’t call `compute()` or `persist()` before it – which sometimes leads to misdiagnosis of what operation is actually causing the error. So, I might try calling `.persist()` before `to_sql` and see if it helps.
And, to confirm, when you say ‘looping through each partition’, do you mean using a for-loop, or using [`map_partitions`](https://docs.dask.org/en/stable/generated/dask.dataframe.map_partitions.html)? I’d suggest checking out `map_partitions` if you’re not using it already!