In our series on hybrid SAP HANA Data Warehouse architectures, we explore the combination possibilities of the three SAP Data Warehouse solutions.
In the first part, we provided an overview of the key characteristics and strengths of the three variants and discussed the general motivation for a hybrid architecture within the SAP DWH context. Subsequently, we delved into the possibilities for jointly utilizing the established products SAP BW/4HANA and SAP HANA, highlighting their advantages and challenges.
Even within the scenario presented in the first part, we were able to incorporate SAP's latest Data Warehouse product, Datasphere, to facilitate the user-friendly integration of a data warehousing approach that equally relies on BW/4 and a HANA-native SQL DWH.
In the second part of this series, we revisit this concept and outline methods for establishing a modern and future-proof Enterprise Data Warehouse based on the open SQL approach and Datasphere.
Table of Contents
1 New Flagship Datasphere – Why Still Hybrid?
1.1 Datasphere Introduction Phase
1.2 Current Functional Focus on Business Users and Self-Service
1.3 SAP Emphasizes the Benefits of Hybrid Strategies
2 Datasphere and HANA SQL – A Perfect Pair?
2.3 Agile Development Process and Lifecycle Management
New Flagship Datasphere – Why Still Hybrid?
Datasphere Introduction Phase
SAP Datasphere has been available since late 2019 and is still in its go-to-market phase. SAP continues to drive sales, and initial customer references are already available. However, a massive surge in adoption does not yet appear to be materializing. While we observe noticeable interest in our consulting practice, the product's capabilities and its specific use cases still raise many questions that, due to a lack of proven practical examples, can currently only be answered predominantly theoretically. Furthermore, approximately 120 product enhancements are already planned for the coming year (see https://roadmaps.sap.com/ for Datasphere). As an Early Adopter (see Figure 2), you would have the opportunity in this phase to familiarize yourself with the application and cloud structures, and potentially benefit from influencing important product adaptations and participating in improvements. However, large-scale deployment in this phase only makes sense if the current functional offering precisely meets your requirements. We will elaborate on this in the next paragraph.
Current Functional Focus on Business Users and Self-Service
At present, Datasphere primarily focuses on business users from specialist departments / lines of business. These users are intended to be enabled to independently build and manage the data structures required for their individual analyses, largely independent of central IT and isolated from other departments. To achieve this, Datasphere provides a container concept of so-called Spaces, where specialist departments find isolated working environments to persist data and, in particular, to process it virtually. Data exchange between these Spaces is nevertheless possible. As a best practice, SAP envisions, for example, one or more centrally IT-managed Spaces where data from various source systems is ingested and then distributed to the departmental Spaces. This approach aims to achieve an optimal balance between central data governance and individual self-service. For data preparation, the DWH applications offer two essential layers: the Data Layer and the Business Layer (Figure 3), which we will only briefly touch upon here. More detailed explanations can be found in our book SQL Data Warehousing with SAP HANA read more in a chapter dedicated to Datasphere.
The Data Layer is designed for the backend preparation of the Data Warehouse. Here, data can be ingested, transformed, and integrated using SQL functions. From our perspective, this area is still significantly underdeveloped compared to established solutions like BW/4HANA and HANA SQL. Mechanisms for data historization, capabilities for implementing complex logic, and control functions for data loading are currently notably absent. This is further underscored by a review of the planned product enhancements, nearly half of which focus on data integration, data modeling, and data provisioning, and include core functionalities such as connectivity options to PostgreSQL, Teradata, and MySQL, Window Functions, and scheduling functions (see https://roadmaps.sap.com/ for Datasphere).
Building upon the Data Layer, the Business Layer facilitates the semantic modeling of dimensions and facts. Here, business users are empowered, through graphical tools or SQL code, to independently prepare data according to their specific requirements for a consuming frontend. In our assessment, the functionalities at this level are not yet as mature as those in BW/4 or the native HANA SQL approach. However, we consider the current state suitable for self-service scenarios, and the roadmap consistently promises new functionalities in this area.
A significant advantage we observe is the direct integration of SAP Analytics Cloud as a data visualization frontend, enabling semantic modeling to support this third layer very effectively. Furthermore, suitable interfaces are intended to facilitate the seamless use of non-SAP frontends.
Overall, an examination of the functional scope reveals that Datasphere currently emphasizes the business-user-centric layers of a data warehouse application, leveraging its strengths in this domain. The application will only evolve into a full-fledged EDWH solution in the coming years, as the following specific aspects require development to achieve enterprise readiness:
- Modeling
Business-driven modeling, independent of the modeling method, with a clear separation between design time and runtime. - Data Integration
Diverse connectors to key sources (SAPI, REST, Kafka), complex transformations, complex orchestration. - Agile Development Process for Large Developer Teams
Capabilities for parallel development in larger developer teams through sandboxing, version control, and DevOps-oriented Continuous Integration / Continuous Delivery. - Lifecycle Management
Development and operations according to DevOps standards with Infrastructure as Code and automated pipelines for simple and continuous evolution. - Operations
Job Handling and Monitoring. - User & Authorization Management
Granular authorization on structures and data content.
SAP Emphasizes the Benefits of Hybrid Strategies
It is therefore no coincidence that SAP openly promotes hybrid constellations with existing DWH solutions such as BW/4HANA and HANA SQL. While the emphasis is primarily on connecting on-premises and cloud products, aiming to illustrate and encourage a gradual transition to the cloud, SAP should implicitly recognize that Datasphere, especially when compared to the two established SAP DWH products, does not yet represent a full-fledged EDWH solution. Too many core functionalities are still missing and must be accommodated in hybrid scenarios.
This situation is further underscored by the very long-term assurance of BW/4HANA support until 2040, and the interoperability with BW currently being diligently advanced by SAP through SAP BW Bridge.
We also believe that, in the coming years, while considering Datasphere, you should not lose sight of a mature EDWH solution. BW/4HANA and native HANA SQL Data Warehousing offer excellent options for this. In the second part of this blog, we will illustrate the arguments in favor of a synergy between Datasphere and HANA SQL DWH.
Datasphere and HANA SQL – A Perfect Match?
In recent years, behavioral biology has increasingly demonstrated that the folk wisdom 'birds of a feather flock together' can indeed be substantiated concerning personality structures for social interaction and successful relationships (see, for example, https://www.vbio.de/aktuelles/wissenschaft/gleich-und-gleich-gesellt-sich-gern/, https://www.tagblatt-wienerzeitung.at/nachrichten/wissen/mensch/2136808-Soziale-Stabilitaet-laesst-sich-mit-einfacher-Regel-erklaeren.html). This information might seem unusual in a technical blog about SAP software. However, we also consider this thesis a compelling argument when evaluating a promising hybrid architecture with SAP Datasphere.
While we do not intend to attribute a personality structure to Datasphere and other DWH solutions, we nonetheless focus on their essential inherent values, albeit from a technical perspective. In this context, we observe that Datasphere is a HANA application running on SAP's Cloud Platform Business Technology Platform (BTP). Compared to BW, the historical SAP ABAP stack is absent. Its technical structure is characterized by the interplay of the SAP HANA database and Cloud Foundry, an environment for developing and operating cloud applications in various programming languages (https://www.cloudfoundry.org/).
Datasphere and the open SQL approach are, in this regard, cut from the same cloth and are congruent, especially concerning database development essential for data warehousing. In the coming years, Datasphere is primarily aimed at bringing the existing data warehousing functionalities of the HANA or Business Technology Platform, based on this foundation, into a more or less closed application context that guides developers and business users and prescribes clear solution paths with simple design tools.
With a hybrid alignment based on Datasphere and HANA SQL, you therefore benefit from numerous advantages. You can already begin to re-establish aging DWH structures on HANA on-premises or in the cloud, or process only those parts requiring an individualized approach within the HANA SQL DWH.
Concurrently, you can familiarize yourself with Datasphere's capabilities and better integrate your business users into your enterprise Business Intelligence initiatives through enhanced self-service. Furthermore, the agile and model-driven development capabilities of the HANA SQL approach, incorporating DevOps principles, enable you to rapidly and flexibly integrate analytical value into your business processes and enhance your productivity.
In the long term, you will likely be able to cover significant portions of your entire DWH structure with Datasphere, as all necessary EDWH functionalities will be available within a few years. However, HANA SQL Data Warehousing can still provide valuable service at this stage, as it will likely continue to better support areas of your data warehousing processes that require highly individualized solutions.
Overall, the following aspects of this interplay can be particularly highlighted.
EDWH Modeling
Datasphere's native capabilities do not provide sufficient options for business-driven, flexible modeling that significantly facilitates the comprehension of complex business logic. However, you can offload such a modeling approach to SAP PowerDesigner and graphically design your entire EDWH structures there. You can create the Line of Business models within Datasphere, thereby continuing to benefit from a clear separation between departmental content and centrally IT-managed content.
A crucial aspect is that you can directly execute the DDL statements generated by PowerDesigner, based on your graphical data models, within Datasphere's EDWH space, thereby transferring the outcome of the modeling process directly into Datasphere. This is achieved by accessing the underlying Business Technology Platform and the corresponding HANA database (Figure 4).
This approach enables you to utilize, for instance, a Data Vault model for the EDWH component, which would not be feasible using Datasphere's native functionalities. Furthermore, Datasphere's inherently open architecture allows you the flexibility to employ other non-SAP software, such as the ERWin modeling tool, if desired.
Data Integration
Analogous to modeling, you can proceed similarly in the domain of data integration. Datasphere's open architecture, built upon BTP and SAP HANA, enables you to utilize a data integration tool of your choice to populate Datasphere's EDWH Space with data. As Datasphere's underlying foundation, the Business Technology Platform (BTP) inherently includes the Smart Data Integration (SDI) package (Figure 5). Data Intelligence is another compatible SAP product. However, this approach is fundamentally compatible with any other ETL tool as well.
Agile Development Process and Lifecycle Management
Another significant aspect that you can fully leverage in the interplay between Datasphere and the underlying Business Technology Platform is an agile development approach and sustainable lifecycle management. Within the EDWH component, you can ensure development quality and prevent production outages by utilizing various environments.
Through Continuous Integration / Continuous Delivery processes, based on version control systems and automation servers like Git and Jenkins, you can establish granular transport pipelines and rapidly and continuously generate productive added value in a DevOps style. The Data Warehouse for the various Lines of Business remains unaffected, with its structures merely accessing the IT-required source formats provided within the EDWH Space (Figure 6).
Separate or Under One Roof?
In line with our earlier analogy, the question ultimately arises as to how you wish to architecturally arrange the interplay between the EDWH component, based on the Business Technology Platform, and Datasphere. The preceding figures suggested a separation where Datasphere's spaces reside on a different tenant than the BTP / HANA EDWH content.
Nevertheless, a solution on a single tenant, thereby consolidated under the Datasphere umbrella, is also feasible and further streamlines the architecture (Figure 7). Overall, this offers numerous possibilities to establish your EDWH either entirely in the cloud or, if required, partially on-premises, deferring the transition to a single tenant until a later point.
Conclusion
Christopher Kampmann
Head of Business Unit
Data & Analytics
christopher.kampmann@isr.de
+49 (0) 151 422 05 448
