Articles & Guides

Altimate Code and the Rise of the AI Engineering Harness

The Hidden Dependency Behind Reliable AI Agents

dbt wizard cli: AI agent built for analytics engineering

dbt Wizard CLI: An AI Agent Built for Analytics Engineering

AI Cost Observability: Two Open Source Tools Every AI Developer Should Know

AI Skills Need CI/CD Too

Stop Guessing Your AI Spend: Two Open Source Tools That Track Every Token

dbt MCP v1.20.0: Your AI Agents was reading the Wrong dbt docs

How dbt MCP v1.20.0 fixes a subtle but important problem in AI-assisted analytics engineeringOverviewdbt MCP v1.20.0 introduces version-aware documentation retr...

Your AI Agent Was Reading the Wrong dbt docs

dbt v1.12 vars.yml: A Five-Year Community Ask, Finally Answered

dbt Core v1.12 Rewrites the Semantic Layer Spec

How I Use Promptfoo to Test and Grade an Agile AI Skill

I treat agile story skills like software. That means I test them, score them, and grade them — using promptfoo — before users hit a regression.Here is why that...

How to Use Promptfoo to Test and Evaluate AI Agent Skills

A practical guide to benchmarking AI skills against baseline agents, measuring quality improvements, and building confidence before production adoption.I treat...

Testing AI Skills Like a Software Engineer

Defining Better Sprint Goals with GitHub Copilot Agent Skills

Using AI-powered problem framing and sprint-goal generation to align agile teams around outcomes instead of task lists.Agent skill GitHub Repo link in reference...

My Copilot Agent Skill Splits Stories Better Than Most Agile Teams

How eight proven story-splitting patterns turn 13-point sprint killers into vertical slices that actually ship value.The agile-story-splitter Copilot Agent Skil...

Why Your User Stories Are Broken (And How a Copilot Agent Skill Can Fix Them)

Injecting Deterministic Requirements, BDD Acceptance Criteria, and Automated Quality Gates Directly into Your BacklogSkills Repository: https://github.com/alwyn...

The Complete Guide to Agent Skills for Agile: Four Skills for Better Scrum

AI Agents Explained Without the Hype: A Data Engineer’s Take

MCP vs. Agentic Skills: Which Architecture Should You Choose?

A Better Way to Do Real-Time Streaming (Redpanda + RisingWave Explained)

BI-as-Code: Bridging the Gap Between Data Engineering and Analytics

BI-as-Code treats dashboards and metrics as version-controlled, testable, reusable code artefacts closing the last remaining gap in the modern data stack where...

MCP vs Agentic Skills — What’s the Difference, and How Do They Work Together?

Understanding where protocol ends and intelligence begins in modern AI agent architectures.MCP (Model Context Protocol) and agentic skills are not competing con...

Databricks transitioning to the Direct Deployment Engine for Declarative Automation Bundles

A “Software Engineering” mindset for data, without the infrastructure tax.The landscape of Databricks automation is shifting. With the release of Databricks CLI...

The dbt MCP Server: How the Model Context Protocol Bridges Governed Data to AI Agents

LLMs are smart but blind. Your semantic layer is organised but silent. MCP is the bridge that lets AI see your data — and answer questions without hallucinating...

Metrics as Code: Building a Semantic Layer With dbt and MetricFlow

How to define business metrics once, version them in Git, and serve them consistently to every dashboard, notebook, and AI agent in your stackThe Idea in One Se...

Why Data Teams Need a Semantic Layer

The Problem: Metric Disagreement Across Tools and TeamsMost data teams have encountered this situation: two reports that should show the same number don’t. Fina...

dbt fusion — Under the Hood — The Technical Architecture

dbt fusion : Under the Hood—The Technical ArchitectureMore Than Just “Faster”The dbt Fusion Engine represents a fundamental architectural evolution from the leg...

The Evolution — Why dbt Needed a Revolution

Speed Up Your dbt Development with Sample Mode

Customizing Your dbt Documentation with node_color

If you have ever looked at your dbt DAG and wished you could add some visual distinction to help your team quickly identify different model types, domains, or d...

Implementing Data Contracts with dbt: From Theory to Practice

Real code, real patterns, real results

The Business Case for Data Contracts: Quantifying ROI

Show your CTO the numbers that justify the investment

Data Contracts: The Missing Link in Your Data Engineering Strategy

Here’s the uncomfortable truth: most data teams spend 60–80% of their time dealing with data quality issues instead of delivering actual business value. That’s...

Shift Left Governance: Why You Are Fixing Data Problems at the Wrong Time

What DevOps taught us about preventing problems instead of reacting to them

The Code Review Journey: From Manual Slog to AI-Powered Efficiency

I will be honest — code reviews used to be the part of my day I would quietly dread. Not because I didn’t value them, but because they felt like pushing a bould...

Building Context-Aware AI for Your Team

In Parts 1 and 2, we explored why code reviews are broken and how context-aware AI fixes them. Now let’s get practical: exactly how do you implement this for yo...

Teaching AI to Understand Your Codebase

In Part 1, we explored why code reviews have become such a bottleneck and why standard AI tools don’t solve the problem. Now let’s talk about what actually work...

From 4 Days to 2 Hours: How Context-Aware AI Transformed Our Code Reviews

The bottleneck you don’t talk about in standup meetings, but everyone feels: code reviews that drag on for days, developers frustrated by wait times, and review...

The Code Review Bottleneck Nobody Talks About

Read the main article in the series before doing a deep dive.

Don’t Nest Your Curlies: A Practical Guide to Jinja in dbt

The most common syntax mistake in dbt projects — and how to avoid it

Top 20 Data Engineering Resources from Tech Giants

Stay up-to-date with the latest innovations, best practices, and technical insights from leading technology companies. These engineering blogs offer invaluable...

Future of Data Engineering: Why Your Ontology Layer Matters More Than Ever

I have been giving a lot of thought to how we design and structure data architecture to effectively support AI initiatives.

Applying MLSecOps to Secure the AI Lifecycle

The proliferation of Artificial Intelligence (AI) and Machine Learning (ML) across enterprise environments necessitates a dedicated, rigorous security strategy....

MLSecOps Implementation and Strategy: Securing the AI Lifecycle

Successfully implementing MLSecOps requires infusing security practices directly into the existing MLOps pipeline. This strategy, built on the pillars of people...

MLSecOps: Securing the Future of AI and Machine Learning

Artificial intelligence is no longer a futuristic concept — it’s embedded in our daily lives. From voice assistants that understand our commands to automated fi...

The MLSecOps Dream Team: Building Cross-Collaborative AI Security

In the age of AI, software security isn’t just about code — it’s about algorithms, data pipelines, and complex models. This shift demands a radical evolution fr...

From Theory to Production: The dbt CI/CD Pipeline Codebase

If you haven’t explored my full dbt CI/CD pipeline series yet, I highly recommend starting with the complete article collection here:

dbt Snapshot vs Custom SCD2 with CDC — Comparative Analysis

Slowly Changing Dimensions (SCD Type 2) are foundational for tracking historical data in analytics and compliance. In modern data stacks, two dominant approache...

Custom SCD Type 2 Model in dbt

Building robust data warehouses that preserve historical context

Why Your AI Projects Are Failing (And It’s Not the AI’s Fault)

Ever feel like your AI projects are hitting a wall, even when the tech seems fine? Often, it’s not the AI itself, but the hidden “trust erosion” in your data th...

How to Refactor a Complex Logic in dbt Models

A common challenge in data engineering is managing complex SQL models with numerous joins and transformations. A single, monolithic model can become a maintenan...

Why Your Data Transformation is Failing

Your company just spent a fortune on new data tools. But if people aren’t using them, was it worth it?

Databricks PII — Identification, Protection, and Management

In today’s>The Growing Challenge of PII ManagementAs businesses migrate to modern data platforms like Databricks, they often discover that PII is scattered acro...

Discovering PII Data in Databricks with discoverx

discoverx, a project from Databricks Labs, is a powerful "Swiss-Army-knife" toolkit designed to automate administrative tasks across your Databricks Lakehouse....

Building the Complete GitHub Actions Workflow

Article 5 of 5: Building a Production-Ready dbt CI/CD Pipeline

Building a Production-Ready dbt CI/CD Pipeline: Complete Series

A comprehensive guide to implementing automated quality checks, formatting, and testing for your dbt projects

Consistent YAML Formatting with Prettier and npm

Article 4 of 5: Building a Production-Ready dbt CI/CD Pipeline

Enforcing SQL Standards with SQLFluff

Article 3 of 5: Building a Production-Ready dbt CI/CD Pipeline

Implementing Pre-commit Hooks for dbt Quality Control

Article 2 of 5: Building a Production-Ready dbt CI/CD Pipeline

Setting Up Your dbt Project for CI/CD Success

Article 1 of 5: Building a Production-Ready dbt CI/CD Pipeline

DevOps to MLOps: Building a Complete Data-to-Model Pipeline

This article breaks down a typical modern ML lifecycle, highlighting the critical interplay between DataOps, which focuses on reliable data delivery, and MLOps,...

Data Product Engineering — Core Principles

In today’s digital world, businesses generate vast amounts of data. But simply having data isn’t enough. To unlock its true potential and generate tangible outc...

Building Data Cards for Data Products Using dbt

Data cards are structured documents that provide essential metadata, quality metrics, lineage information, and usage guidelines for data products. They serve as...

Data Quality and Data Lineage in Data Cards

Let’s face it — bad data creates real problems. When data is messy, outdated, or just plain wrong, it can slow down operations, lead to poor decisions, and cost...

dbt as well run kitchen station

In this article, I use a cooking analogy to explain the different layers of dbt modeling. If you haven’t yet read my article on dbt modeling layers, I encourage...

12 Golden Rules for dbt Success

After working extensively with dbt in production environments across multiple organisations, I have compiled this expanded set of gold standards for dbt that ha...

How to Use the zip function in dbt

The zip function in dbt is a context method that allows you to combine multiple iterables (like lists) into a single iterator of tuples. Each tuple contains ele...

Evolution of Data Architecture: A Comparative Analysis

The journey of data architecture began in the late 1980s with the advent of data warehouses. This period was characterised by on-premises infrastructure and rel...

Understanding Principles of Data Mesh Architecture — part 1

Challenges with Traditional Data Lakes

Understanding Principles of Data Mesh Architecture — part 2

In this section — we will deep dive into each of the guiding principles of data mesh

Use dbt retry to run from point of failure

The dbt retry command is a powerful tool in the dbt (data build tool) ecosystem, designed to re-execute the last dbt command from the point of failure. This com...

dbt Cloud Setup Guide with Databricks

In this article, we will walk through the steps required to set up dbt (data build tool) Cloud using Databricks as your data warehouse. This setup allows you to...

dbt-testgen: Automate Your dbt Test YAML Generation

dbt-testgen is a powerful dbt package designed to automatically generate dbt test YAML files based on real data. Inspired by dbt-codegen, this tool simplifies t...

Ensuring Data Quality with dbt-expectations

In the world of data engineering, ensuring the accuracy and reliability of your data is paramount. Enter dbt-expectations, an extension package for dbt (data bu...

MLOps is Data Engineering

MLOps is the practice of applying DevOps principles to the development and deployment of machine learning models.

Automate Your Python Workflow with zsh-autoswitch-virtualenv

In the world of Python development, managing virtual environments can be a tedious task. Enter zsh-autoswitch-virtualenv, a ZSH plugin designed to streamline th...

Understanding dbt Modelling Layers and Their Purpose

Models — The landing layer, also called the raw zone or source layer, acts as the initial point of entry for data.

dbt-checkpoint — Improve Data Quality

Data is the lifeblood of good decision-making, but it’s like food: if it’s rotten, it can make you sick. That’s where data quality comes in. It’s all about maki...

dbt Model Contracts

dbt Model Contracts is a way to statically define the shape of your data and is defined in a YAML file, specifying the names, data types, and any additional con...

Display current git branch name on Terminal using zsh

Open ~/.zshrc in your favorite editor and add the following content to the bottom.

Using SQLcl Bulk export Oracle table into CSV and load into GCS

In this article I will explain how you can quickly download list of tables from Oracle database into csv format and copy them into google cloud store.

How did I gain 40% reduction of docker image build for dbt databricks

I was recently buiding a docker images for dbt-databricks and even through I used the slim version of the python image the size of the overall docker image was...

How to install dbt Docker for databricks

In this article I will explain you how did I create a docker image for databricks. As of Aug 2023, I have not found a recipe to build dbt-databricks image.