GeoLLM-Bench benchmarks large language models on geospatial programming tasks, including GIS automation, remote sensing, and spatial analysis. It provides task datasets, prompts, evaluation scripts, and baseline results to assess LLM performance in geospatial coding.
Welcome to the companion repository for my Master's thesis exploring how well Large Language Models (LLMs) can navigate the world of geospatial coding! 🧭💻
This little corner of the internet holds the digital artifacts from my research, where I put different AI models (like GPT-4o, Llama 3, DeepSeek, Gemini, and Falcon!) to the test on a range of 100 geospatial programming challenges using Python. Think of it as a map charting the journey of LLMs trying to speak "GIS".
Inside, you'll find:
- 📝 Details about the benchmark tasks used in the study.
- 🐍 Python code snippets generated by the different LLMs for each task.
This repository is organized primarily by LLM model outputs and human evaluation notebooks. You’ll find folders such as:
/gpt-4o-, /gemini-, /llama-, /deepseek-, /falcon-* Model-specific directories containing benchmark data and Python code generated by each large language model for the geospatial programming tasks. Each folder corresponds to a particular model and run.
/HumanEvaluatedCode-* Jupyter notebooks and scripts used for human evaluation of the generated code, including qualitative assessment and performance analysis.
README.md Overview of the project, benchmark goals, and usage instructions.
LICENSE Project licensing information (MIT).
This repository supports my Master's thesis titled "**Benchmarking Large Language Models for Geospatial Programming Tasks
**" completed at NUST, Islamabad. The core goal was to develop and utilize a benchmark framework to objectively assess the capabilities and limitations of modern LLMs in generating functional and efficient Python code for common geospatial operations involving libraries like GeoPandas, Rasterio, GDAL, and more.
Thanks for stopping by and exploring the intersection of AI and Geospatial Science! Feel free to look around. 😊