Elasticsearch Dask Dataframe

[Behzad]

Hi guys,

Hope you all feel well, I’m a newbie in DASK. I have trouble In creating Dataframes with read_sql_query/read_sql_table with Elasticsearch indexes. Normally for an SQL-based table, it’s easy to fetch data as Dask Dataframe directly but in Elasticsearch it’s a little bit vague.

Thank you all.

[guillaumeeb]

Hi @Behzad, welcome here,

I’m not sure what you are asking here. Elasticsearch is not an SQL database, so it feels normal it won’t work out of the box with these methods. Elasticsearch APIs returns JSON documents, so I’ll guess you’ll have to build something on your own.

Doing a quick Internet search, I’ve found several resources:

• A Dask ELK plugin: DaskElasticSearch API — dask-elk 0.1.0 documentation
• A stack overflow answer : python - How to write to Elastic db in Dask? - Stack Overflow
• And ultimately, a description on how to create Pandas Dataframe from Elasticsearch How to Use Elasticsearch Data Using Pandas in Python | ObjectRocket.

Maybe one of those could help you?

Starting from scratch, you should use the Elasticsearch Python client, get a JSON, and create a Dataframe from it, either parsing it, or maybe using Dask Bags.