A Metadata Management System for Apache Wayang

BSc / MSc Thesis.

This project focuses on designing and implementing a metadata management store for Apache Wayang. Wayang lets users express a data processing task in a platform-agnostic way and then decides which underlying platform (e.g., Spark, Flink, Postgres, Java) should execute each part of the plan. Today, however, Wayang has no dedicated component for managing metadata: information such as schemas, statistics, or learned cost functions is handled implicitly by individual subsystems rather than stored and reused in one place. The goal of this project is to add such a component and enable Wayang to store, retrieve, and manage its own metadata in a structured and extensible way.

The proposed metadata store should support different types of metadata, such as table and schema metadata, function (UDF) metadata, cost and profiling information, or access-control–related information. Rather than relying on a fixed set of predefined object types, the system should allow metadata types to be defined dynamically and then store metadata objects that conform to these types. This makes the approach flexible and adaptable to different use cases and future extensions. The metadata itself can be persisted in a relational backend such as Postgres, MySQL, or SQLite.

A metadata component of this kind would allow Wayang to maintain internal state that other Wayang components can build on. Examples include exposing metadata through the Wayang UI, or feeding persisted statistics and learned cost functions back into the optimizer so that plan choices improve over time instead of being recomputed implicitly. It also opens up opportunities for better integration with the surrounding data ecosystem, where metadata catalogs and formats such as Apache Iceberg, Apache Gravitino, and DuckLake have become de-facto interoperability points.

Research questions include:

• What is a suitable data model for a Wayang metadata store that supports dynamically defined metadata types while still allowing objects to be validated and queried efficiently?
• Which categories of metadata (schema, statistics, UDF/function, cost/profiling, access control) matter most in the context of Wayang, and how should they be represented?
• Can persisting metadata, in particular statistics and learned cost functions from Wayang’s profiler, measurably improve the cross-platform optimizer’s plan choices compared to the current implicit handling?
• What are the performance and storage trade-offs of a dedicated metadata store, and does it add noticeable overhead to query planning?
• How does the API of the proposed metadata store compare to that of established metadata catalogs and formats such as Apache Iceberg, Apache Gravitino, and DuckLake, and what can Wayang learn from or reuse of them?

The project combines system building with research. A bachelor’s thesis can focus on designing and implementing a first version of the metadata store and validating its feasibility within Apache Wayang. A master’s thesis can go further, evaluating performance, studying the impact on optimizer decisions, and investigating interoperability with other systems in the Wayang ecosystem. The work is open source and carried out in coordination with the Apache Wayang community.