[DataFrame] what about integration of dataframe in Gama?

[chapuisk]

Matrix structure in Gama cannot be manipulated based on explicit headers, like it is usually the case in structure files such as csv, json and others. Matrices in Gaml are also limited in the number of linked data it can store, only 2 dimension. Hence implementing a DataFrame in Gaml could be a great addition manipulating complex data, as well as connecting with extensive ML libraries. This last point could be a game changer when it comes to simulation analysis or population synthesis in Gama (just to talk about the part I have been working on). Furthermore, I know that most of Gama users are familiar of language such as R or Python, even before they learn Gaml. Giving them the ability to manipulate known complex structure could be a plus.

The potential best option is Apache Spark data frames. What is your opinion about this? Should we go for a new gaml type or expends on already existing matrices?

Best
Kevin

[lesquoyb]

⛏👷 Some update on that

I've been working on integrating the dataframe/dataset from apache spark as you suggested, which seemed like a good idea given how well established it is.
But so far I am not convinced it is the right library for us as it seems that dataframes are highly bound to the whole spark system, which is made for distributed computing. So far (it may change as I gain a better understanding) I was forced to also include around 200MB of jar dependencies to make everything work, which also has an impact on booting time of gama + makes debugging tedious probably because of dynamic class resolving. I can probably overcome most of this if I spend more time on it, but it makes me wonder if it's really worth it.

There are other libraries that we could use as a support for our dataframe implementation but which one to pick will eventually depend on what we really want out of dataframes in gama:

• just a data structure with named columns and rows ?
• something that can conveniently be transformed into a csv/json/xml file or generated from it ?
• an efficient structure for matrix like algorithms ?
• a data structure able to handle a huge amount of data without a big cost for the user ?
• a data structure that comes with a library of machine learning algorithms ?
• something that can be used to efficiently/conveniently display a large amount of data ?
• efficiently/conveniently filtering/joining/searching tables, with operators or sql-like syntax ?
• Having an n-dimensional typed data-structure ? This is not really what dataframes are for but you mention it briefly

If you are still interested by that maybe we can take some time to talk a bit about use cases and concrete integration into gama (together with @AlexisDrogoul and whoever feels interested too ?).

[chapuisk]

I am really happy to see that you take this quite seriously!!! I am more than willing to join on a discussion about potential use of such structure, in order to choose best fits to the needs. I think that potential use could target works done by @benoitgaudou (DL based agent and proxy models) and @ptaillandier (graphs); although I can come with concrete use and models.

[lesquoyb]

It took some time but it is finally merged into gama and properly tested. The use cases it serves are:

• A tabular data structure with heterogeneous named-columns (as this is the bare minimum)
• Easily readable/savable to json, csv, xml, xlsx, parquet and avro format
• Able to handle millions of records "efficiently"
• Provide convenient data manipulation operations (filtering, sorting, merging etc.)
  All that backed by the DFLib library