Enterprise Lakehouse Architecture Driven by Metadata for Scalable Data Governance and Advanced Analytics on Azure Databricks

Abstract

The growing need for enterprise data is creating challenges while fulfilling the necessity of effective and scalable systems to store, govern, analyse, and manage data at scale. Conventional data warehouses and data lakes encounter numerous impediments, and poor metadata management can significantly undermine analytical accuracy and governance efficiency. This paper aims to present a metadata-driven enterprise lakehouse architecture that will allow for achieving scalable data governance and advanced analytics in Azure Databricks. The qualitative study relies on an inductive approach and utilises secondary data collected from scholarly articles, peer-reviewed works, and industry-specific reports to demonstrate the benefits of implementing a centralised catalogue, automating data ingestion, optimising data lake architecture, and establishing a metadata-driven data quality framework. The review highlights the ability of Azure Databricks to offer enterprise-grade analytical performance via Apache Spark processing, Delta Lake’s atomicity, concurrency, and schema management, query optimisation, and autoscaling to ensure reliable decision-making. The paper's key research results include the increased importance of metadata management in enterprises due to the data volume explosion. Indeed, the global data volume is predicted to reach 175 zettabytes by 2025. Furthermore, the metadata management tools market size is expected to grow from USD 6.3 billion in 2021 to USD 15.1 billion in 2026. The study also revealed barriers to metadata management such as legacy system integration, data interoperability, governance standardisation, and multicloud management, and recommended further research on artificial intelligence-enabled metadata discovery and lineage, data quality governance, and predictive analytics to drive enterprise lakehouse innovation and sustainability in the era of clouds.