Managing Experimental Data with DuckDB

Observing how well data-intensive systems utilize hardware resources is a crucial preliminary step to improve system performance and reduce hardware waste. To do such observations, one has to collect a lot of monitoring data on hardware behavior through experiments. In our group, we have recently built a framework to aid the management of such monitoring data efficiently, called Resource-Aware Data Systems Tracker. Currently, this framework uses Postgres data management system to keep the experimental data. We would like to checkout alternatives to Postgres in our design. More specifically, test out the impact of using DuckDB instead, as it offers a leaner system design for data analytics and visualization of data.

This project would be suitable as a standalone project or BSc or MSc thesis at ITU during Spring 2024. If you are interested in data management systems, scalable data management, and benchmarking in general, this project would be a great fit for you.