Essential components for building a functioning data engineering practice relevant to the current day at scale. — As data continues to grow in importance and become more complex, the need for skilled data engineers has never been greater. But what is data engineering, and why is it so important? In this blog post, we will discuss the essential components of a functioning data engineering practice and why…

10 Rules for Effective Dimensional Modeling for Data in the cloud — In this blog post, I’ll share the modern data modeling concepts that I’ve implemented as standards across my former team of data engineers and modelers at a Fortune 200 enterprise. Specifically, I’ll delve into the roles of data modeler and data engineer in designing a physical database, while addressing…

Comparing Snowflake vs AWS RedShift vs Google BigQuery Jan 2022 The cloud has changed the way we think about data. No longer are we tethered to physical infrastructure and expensive hardware. Now, we can spin up a data platform or warehouse in minutes, and pay only for what we use…

Databricks >> Delta-Lake >> Delta-Sharing : Setup & Review — What is Databricks? A data management and processing platform provided by the creators of Apache Spark. Shares the same name as the company ‘Databricks’. Built on top of the Apache Spark engine with a unified processing interface across all platforms. Delta Sharing Conceptually Delta Sharing as a service is to allow data to…

Before you invest in Confluent $CFLT IPO, you might want to learn about Kafka. ;) — Apache Kafka common benefits are building data pipelines, leveraging real-time data streams, enabling operational metrics, and data integration across countless data sources. 11 Step technical architecture for building Kafka based DataStreams: Cluster Reference Architecture A Confluent Platform cluster that is built for high-throughput long-term scalability will have an architecture similar to the following:

4 Data Security services available for securing data on Snowflake. — Sensitive Data needs to be organized, protected, and used. Enterprises needing additional user level security for governing sensitive PII data within Snowflakes Data Cloud need to resort to plenty of scripting to manage the roles, users & policies within Snowflake or use one of the many vendor services available…

The differences between an ETL/ELT tool vs an iPaaS are below.

Storing un-structured Image files in Snowflake database from Azure blog. This post lists end-to-end process for viewing web hosted images in Tableau in 6 easy steps. This process involves integration between Azure blob storage and Snowflake database. Below are high level step’s involved. 1) Create external BLOB storage to save image files in Azure cloud. 2) Create cloud storage integration…

Server & Client setups. — Install BOTH Snowflake ODBC drivers. Don’t change any install paths, these must go where the install says. This will appear for all the copy and renaming. Click Continue and then Yes on the next screen.