SME Tech Stacks

In essence, there are three tiers or layers to any business intelligence ecosystem. Bottom-up, they are the Sources/Input Layer, the Aggregation/Transformation Layer (traditionally known as The “ETL Layer”), and the Visualization/Reporting Layer.

Though core layers (data sources, data management, ETL, reporting/visualization) are familiar, optimizing their synergy and individual approach significantly impacts the resulting BI ecosystem's capabilities. Holistic data management, crucial for automation and real-time reporting, often gets overlooked. While visualization might grab initial attention, advanced data solutions that have equal focus across all layers result in a thriving BI ecosystem in today's dynamic landscape.

Frequently BI projects focus on the Visualization/Reporting aspects but place less emphasis on the layers that underpin this. All layers need equal focus to deliver a successful BI Ecosystem in today’s ever-evolving environment.

SME has mapped out three example tech stacks that our customers use most frequently:

• Data Lakehouse Architecture
• Data Warehouse Architecture
• Azure Data Architecture

Data Lakehouse Architecture

SME's lakehouse architecture is customizable to fit your already existing cloud data lake, like Azure Data Lake or Amazon S3. We utilize tools that seamlessly, and automatically, fill your data lake directly from your data sources.  Once the data is in the lake, a data lakehouse is installed to "sit on top of" that data lake or federated data sources. The data lakehouse serves as a query engine, semantic layer, and data catalog for use in your business intelligence and data analytics tools, like Power BI. 

With a high-performance execution engine, powered by Apache Arrow, speed up queries by 100x and eliminate >90% of compute costs.

This solution looks to solve common pain points like:

• High infrastructure costs from having multiple copies of the data. 
• No flexibility with it comes to the architecture and having to change it every time a new tool is added.
• Legacy data warehouse is slow and causing our bottlenecks in our analytics. 

Learn more about the Data Lakehouse Architecture.

Data Warehouse Architecture

SME's warehouse architecture is designed for organizations needing to consolidate their data into a scalable warehouse that they are able to connect to their analysis tools, like ThoughtSpot. For the data integration piece, SME utilizes a tool designed to connect to disparate data sources. This architecture includes the ability to perform data transformations inside the data warehouse with low code and no code methods. An agile data governance approach is taken, including a data catalog, data dictionary, and business glossaries. 

This solution looks to solve common pain points like:

• Current warehouse is too slow and we pay for more compute than we use.
• Too many data sources organized. 
• Database Architect's already know SQL and we don't want them to have to spend the time learning a new language. 

Learn more about the Data Warehouse Architecture.

Azure Data Architecture

SME's Azure architecture uses a wide variety of Microsoft solutions to analyze data or prepare it for reporting, machine learning, and data science. This architecture utilizes Azure Data Factory, Azure Data Lake Storage, Azure Synapse for the analytics workspace, and Databricks for data science and machine learning. The Azure Data Catalog is implemented to connect to the data lake or Azure data systems to catalog your data/metadata and add a governance layer. The data can then be analyzed in a Business Intelligence (BI) tool like Power BI.

This solution looks to solve common pain points like:

• Not using Azure properly or to it's full ability, minimizing the return on investment. 
• Have Azure Data Factory but not using it.
• Do not know how to incorporate data science using Synapse and Databricks.

Learn more about the Azure Data Architecture.