If you ever sat in a data interview and heard “walk me through your star schema,” you already know the feeling. You can explain facts and dimensions in theory. But the next question is often: how did you get from a messy source to something people can actually query?

This post is my attempt to bridge that gap without drowning you in slides. We will use real JSON from the public SpaceX API, load it into DuckDB on your laptop, and build a tiny star schema. Then we will play with slowly changing dimensions (SCD) — Type 1, 2, and 3 — because interviewers still love that topic, and because it is one of those ideas that only clicks when you run the SQL yourself.

If you didn’t try DuckDB yet, check our lesson Just enough DuckDB for Data Analyst | Module 2.7

No cloud account required. Just DuckDB and a network connection for the first fetch.

Why bother with dimensional modelling?

Here is the honest version. Not every team models everything as a star schema. You will see wide tables, one big denormalized export, or a lakehouse with a dozen patterns at once. Fair enough.

But dimensional modelling is still the language a lot of BI tools, metrics layers, and hiring loops speak. When it fits, it fits well: you separate what happened (facts) from what we call things (dimensions), you pick a clear grain, and you stop arguing about whether “one row” means a launch or half a launch.

In practice you usually end up with:

• Fact tables — events or measurements (one row per launch, one row per order line)

• Dimension tables — descriptions you join to facts (date, rocket, customer)

A simple star schema — fact in the middle, dimensions around it — covers a large share of reporting work. It also shows up in interviews next to grain, keys, and SCDs. This post is not a promise that stars solve every problem. It is a practical default you can explain, sketch on a whiteboard, and back up with a file on disk.

A minute of history, then the words that matter

History in one breath: a lot of what people mean by “dimensional modelling” comes from Ralph Kimball and the books around The Data Warehouse Toolkit.

The pitch was simple: make tables that business people can name, and joins that analysts can repeat without a map of fifty normalized tables.

You will still meet Inmon-style warehouses and hybrid setups. That is fine. In many analytics shops, Kimball-style stars stayed popular because they match how reporting tools think.

Definitions (keep this table nearby; it saves you in interviews):

If you remember only one line for the room: grain first. Everything else gets easier after that.

From “what do we want to know?” to DDL

You do not need a perfect methodology. You need a repeatable path. This is the one I still use when I start from zero.

1) Business questions

Before you name tables, write questions in plain language. Five is enough. Ten is great.

Example:

• How many launches per month?

• What is the success rate by rocket name?

• Which rocket flew most often in this sample?

If you cannot phrase the questions, your grain is probably fuzzy. That is a feature, not a bug — fix it early.

2) Bus matrix (simple version)

A bus matrix is a small grid: facts (business processes) on one axis, dimensions on the other. It looks almost too simple. It is still useful because it forces you to ask: what do we measure, and what describes it?

If you add more facts later (payload, booster reuse, whatever), the win is conformed dimensions: the same dim_date, the same dim_rocket, reused across facts. That is how analytics stops turning into twenty different definitions of “rocket.”

3) Logical design (entities, relationships, attributes)

This is the step people often skip in blog posts. They jump from questions to dim_ / fct_ tables. That works if you already think in stars. If you do not, add one short logical pass so the physical names are not magic.

What “logical” means here: you agree on what exists in the business, how things connect, and what belongs on the fact row — still on paper or a whiteboard. You can stay away from SQL types, indexes, and VARCHAR(200) for a moment.

Grain (again, on purpose): one row = one launch attempt (one mission event in our sample).

Entities:

Relationships (plain language):

• Each launch happens on one calendar day and uses one rocket.

• Many launches can share the same rocket. That is a many-to-one from launch → rocket, and launch → date.

Tiny picture of the join path (star):

4) Physical model (DDL)

Now you map the logical picture to warehouse tables:

Naming is a convention: fct_ for facts, dim_ for dimensions. The important part is that everyone on the team reads the same diagram.

At some point the whiteboard has to become SQL. Below we use DuckDB and a local database file so you can poke at the data without spinning up a warehouse.

DuckDB: fetch launches like you mean it

I like DuckDB for posts like this because the feedback loop is fast. You run SQL, you see rows, you fix your mistake, you move on.

Install DuckDB (CLI is enough). From this folder, create a small database and load 20 recent launches with the helper script.

Create the new file fetch_launches.sql:

-- SpaceX public API (community-maintained). Requires network.
-- Run from this folder:
--   duckdb spacex_modelling.duckdb < fetch_launches.sql

INSTALL httpfs;
LOAD httpfs;

-- Remote JSON can change between HTTP reads; DuckDB may error on ETag mismatch.
SET unsafe_disable_etag_checks = true;

DROP TABLE IF EXISTS stg_launches;

CREATE TABLE stg_launches AS
WITH l AS (
    SELECT *
    FROM read_json_auto('https://api.spacexdata.com/v4/launches')
),
r AS (
    SELECT *
    FROM read_json_auto('https://api.spacexdata.com/v4/rockets')
)
SELECT
    l.id AS launch_id,
    l.flight_number,
    l.name AS mission_name,
    CAST(l.date_utc AS TIMESTAMP) AS launch_ts,
    CAST(CAST(l.date_utc AS TIMESTAMP) AS DATE) AS launch_date,
    l.success AS launch_success,
    l.rocket AS rocket_id,
    r.name AS rocket_name,
    r.type AS rocket_type
FROM l
INNER JOIN r ON l.rocket = r.id
WHERE l.date_utc IS NOT NULL
ORDER BY launch_ts DESC
LIMIT 20;

SELECT 'stg_launches rows' AS check_name, COUNT(*) AS n FROM stg_launches;

Learn more about SpaceX API and available data at

https://docs.spacexdata.com/#intro

What fetch_launches.sql does under the hood:

1. Loads the httpfs extension so DuckDB can read remote URLs.

2. Pulls launches and rockets as JSON from the SpaceX API.

3. Joins them and keeps the 20 most recent launches that have a date.

Run it:

duckdb spacex_modelling.duckdb < fetch_launches.sql

┌───────────────────┬───────┐
│    check_name     │   n   │
│      varchar      │ int64 │
├───────────────────┼───────┤
│ stg_launches rows │  20   │
└───────────────────┴───────┘

Why SpaceX? The data is small, relatable, and a bit fun. Rockets beat generic “customer_id” examples when your brain is already tired after a long day.

Reality check: the API is public and community-maintained. Data can shift. On top of that, DuckDB may complain about ETag changes when it reads the same URL twice. The script sets unsafe_disable_etag_checks = true so you can focus on modelling instead of HTTP trivia. If you want stricter behaviour later, read the DuckDB docs for httpfs and decide what fits your pipeline.

Build the star schema (SQL)

Open the database:

duckdb spacex_modelling.duckdb

We can quickly observe the data with simple SQL SELECT * FROM stg_launches LIMIT 10;

Let’s create our dimensions and facts to help us answer business questions. Run:

-- Calendar dimension from the launch dates in the sample
CREATE OR REPLACE TABLE dim_date AS
SELECT DISTINCT
    CAST(strftime(launch_date, '%Y%m%d') AS INTEGER) AS date_key,
    launch_date AS full_date,
    year(launch_date) AS year_nbr,
    month(launch_date) AS month_nbr,
    day(launch_date) AS day_of_month
FROM stg_launches;

-- One row per rocket (in this sample)
CREATE OR REPLACE TABLE dim_rocket AS
SELECT DISTINCT
    rocket_id,
    rocket_name,
    rocket_type
FROM stg_launches;

-- Fact: one row per launch
CREATE OR REPLACE TABLE fct_launches AS
SELECT
    s.launch_id,
    s.flight_number,
    s.mission_name,
    CAST(strftime(s.launch_date, '%Y%m%d') AS INTEGER) AS date_key,
    s.rocket_id,
    CAST(COALESCE(s.launch_success, FALSE) AS INTEGER) AS success_flag
FROM stg_launches s;

At this point, you have something you can show: a fact table with keys, dimensions you can join, and a date table that makes time rollups boring in a good way.

Sanity-check queries (they also feel good in a portfolio walkthrough):

-- Launches per month in this 20-row sample
SELECT d.year_nbr, d.month_nbr, COUNT(*) AS launches
FROM fct_launches f
JOIN dim_date d ON f.date_key = d.date_key
GROUP BY 1, 2
ORDER BY 1, 2;

-- Success rate by rocket name
SELECT r.rocket_name,
       SUM(f.success_flag) AS successes,
       COUNT(*) AS launches
FROM fct_launches f
JOIN dim_rocket r ON f.rocket_id = r.rocket_id
GROUP BY 1
ORDER BY launches DESC;

SCD: the part where dimensions refuse to sit still

Facts are jealous of attention, but dimensions do the quiet damage. A rocket name in a catalogue can change. A product category can move. A customer tier can be renamed for marketing reasons.

The API gives you today’s label. In a real warehouse, you often need a policy: do we overwrite, do we keep history, do we keep one “previous” value?

We will use one rocket as the story — Falcon 9 in the API — because it shows up in the sample, and everyone recognizes the name.

Falcon 9 id in the SpaceX API (stable id):

5e9d0d95eda69973a809d1ec

Imagine the business renames the rocket from Falcon 9 to Falcon 9 Block 5 on 2026-03-25. The launch facts do not move. Only the dimension does. That is the whole point of SCD: the fact table stayed true when the launch happened. The label is a separate problem.

SCD Type 1 — overwrite

Idea: one row per rocket. New value replaces the old value. History is lost. Fast, simple, and sometimes exactly what the business wants (“we only care what it says now”).

CREATE OR REPLACE TABLE dim_rocket_scd1 AS
SELECT * FROM dim_rocket;

UPDATE dim_rocket_scd1
SET rocket_name = 'Falcon 9 Block 5'
WHERE rocket_id = '5e9d0d95eda69973a809d1ec';

Interview line: “Type 1 is good when the business only cares about the current label.”

SCD Type 2 — full history

Idea: multiple rows for the same natural key (rocket_id). You track valid_from, valid_to, and is_current (or equivalent). Facts join to the version that matches the launch date.

This is the pattern people draw on the whiteboard when someone says “as-of reporting.”

CREATE OR REPLACE TABLE dim_rocket_scd2 (
    rocket_sk BIGINT PRIMARY KEY,
    rocket_id VARCHAR NOT NULL,
    rocket_name VARCHAR NOT NULL,
    rocket_type VARCHAR,
    valid_from DATE NOT NULL,
    valid_to DATE NOT NULL,
    is_current BOOLEAN NOT NULL
);

INSERT INTO dim_rocket_scd2
SELECT
    ROW_NUMBER() OVER (ORDER BY rocket_id) AS rocket_sk,
    rocket_id,
    rocket_name,
    rocket_type,
    DATE '2010-01-01' AS valid_from,
    DATE '9999-12-31' AS valid_to,
    TRUE AS is_current
FROM dim_rocket;

-- Close the old row and open a new version for the rename
UPDATE dim_rocket_scd2
SET valid_to = DATE '2026-03-24', is_current = FALSE
WHERE rocket_id = '5e9d0d95eda69973a809d1ec'
  AND is_current = TRUE;

INSERT INTO dim_rocket_scd2 (rocket_sk, rocket_id, rocket_name, rocket_type, valid_from, valid_to, is_current)
SELECT
    COALESCE((SELECT MAX(rocket_sk) FROM dim_rocket_scd2), 0) + 1,
    '5e9d0d95eda69973a809d1ec',
    'Falcon 9 Block 5',
    rocket_type,
    DATE '2026-03-25',
    DATE '9999-12-31',
    TRUE
FROM dim_rocket
WHERE rocket_id = '5e9d0d95eda69973a809d1ec'
LIMIT 1;

Interview line: “Type 2 is the default answer when you must report as-of history.”

SCD Type 3 — current + previous columns

Idea: keep one row per dimension key, add previous and current columns. You only store one step of history. That sounds limited — it is — but it is easy to explain and sometimes enough for a dashboard footnote (“what did we call it last quarter?”).

CREATE OR REPLACE TABLE dim_rocket_scd3 (
    rocket_id VARCHAR PRIMARY KEY,
    rocket_name_current VARCHAR NOT NULL,
    rocket_name_previous VARCHAR,
    name_change_effective_date DATE
);

INSERT INTO dim_rocket_scd3
SELECT
    rocket_id,
    rocket_name,
    NULL,
    NULL
FROM dim_rocket;

UPDATE dim_rocket_scd3
SET rocket_name_previous = rocket_name_current,
    rocket_name_current = 'Falcon 9 Block 5',
    name_change_effective_date = DATE '2026-03-25'
WHERE rocket_id = '5e9d0d95eda69973a809d1ec';

Interview line: “Type 3 is rare. It is useful when you only need one prior value, not full history.”

Quick comparison

What I hope you take away

1. Start from questions and grain. If that part is clear, the rest is mostly typing.

2. A bus matrix is a small tool that saves you from building twelve different “rocket” concepts.

3. Logical design (entities, relationships, attributes) is the bridge between the matrix and dim_ / fct_. Skip it only when the team already shares the same picture.

4. DuckDB + httpfs is a nice way to pull JSON from an API and get back to SQL fast.

5. SCD is not magic. It is a policy for how dimension changes meet historical facts — Type 1, 2, and 3 are the usual interview set for a reason.

If you run the scripts in this post, you will have more than slides. You will have a file you can open, query, and break on purpose. That is still one of the best ways to learn.

Resources

• SpaceX API — community API for the data we use.

• DuckDB httpfs — reading remote files.

• The Data Warehouse Toolkit — Ralph Kimball and Margy Ross — a deeper reference on dimensional modelling.

It’s called Claude Code, and it runs right in your terminal.

What is Claude Code?

Claude Code is an AI coding agent built by Anthropic, the company behind the Claude family of large language models. Unlike a chatbot that gives you code snippets to copy-paste, or an autocomplete tool that guesses your next line, Claude Code has full access to your project. It can read your files, edit them, run commands, work with git, search the internet, and manage entire projects — all from the terminal.

Think of it as a very capable junior developer sitting next to you, except it never gets tired and it can process your entire codebase in seconds.

# Install Claude Code
curl -fsSL https://claude.ai/install.sh | bash

# Start it in your project
cd my-project
claude

That’s it. You’re now talking to an AI agent that understands your project structure, your code, your dependencies — everything.

Why is it so popular?

Three reasons:

1. It actually does things, not just suggests them. When you tell Claude Code to “add error handling to the data pipeline,” it reads the relevant files, writes the code, and can even run the tests to verify it works. Other tools give you a code block to copy. Claude Code makes the changes directly.

2. It fits into how data professionals already work. We live in terminals. We run Python scripts, SQL queries, dbt commands, Spark jobs. Claude Code meets us where we are — no new IDE to learn, no browser tab to switch to. Just your terminal.

3. The quality is genuinely good. Anthropic’s Claude models consistently rank at the top of coding benchmarks (SWE-bench). Claude 4 Opus solves ~72% of real-world GitHub issues autonomously. That’s not a toy — that’s a tool you can rely on for production work.

A Quick Primer: How LLMs Power All of This

To understand why Claude Code works the way it does, it helps to know the basics of what’s under the hood.

Large Language Models (LLMs) are programs trained on billions of texts. They learn patterns — which words follow which, how code is structured, what logic connects ideas. When you ask a question, the model doesn’t look up an answer in a database. It generates text token by token, predicting the most likely next word based on everything it’s learned.

A few key concepts:

• Tokens — LLMs work with chunks of text called tokens. “Hello” is 1 token. “authentication” is 3-4 tokens. Every model has a context limit — the max tokens it can process at once.

• Context window — Claude’s context window is 200K tokens (~500 pages). That’s enough to hold your entire project in “memory” during a conversation.

• The agentic loop — Claude Code doesn’t just answer once and stop. It works in a cycle: gather context (read files), take action (edit code, run commands), verify results (run tests), and repeat until the task is done.

This is what makes Claude Code an agent rather than just a chatbot. It autonomously decides what to do next, uses tools, and iterates on its own work.

What can you actually do with it?

Here are some real scenarios from my daily work:

Analyze data:

> Download Microsoft stock data for the last year.

Plot price with MA50 and MA200. Add trading volume below.

Mark key events on the chart.

Claude Code creates the Python script, installs dependencies, generates the chart, and saves it as a PNG. In about 2 minutes.

Build an API:

> Create a REST API on FastAPI for managing notes.

CRUD operations, SQLite via SQLAlchemy, Dockerfile included.

Debug errors:

> I get “TypeError: cannot unpack non-iterable NoneType object”

in processor.py line 45. Fix it.

It reads the file, finds the bug, fixes it, and runs the tests. If the tests fail, it fixes those too.

Work with git:

> Look at what changed, commit with a good description, and create a PR.

Run multiple agents in parallel:

> Create a team: one agent handles the backend, one writes tests,

one builds documentation. Coordinate through a shared task list.

Yes, you can run an army of agents — multiple Claude Code instances working simultaneously on different parts of your project. This is where things get really powerful.

What I built: Claude Code 101 for Data Professionals

I’ve put together a free course that covers everything from LLM basics to running coordinated agent teams. It’s written specifically for data engineers, analysts, and BI developers — not web developers, not mobile developers, us.

10 modules, from zero to advanced.

What you’ll learn

1. LLM Fundamentals - How models work, tokens, context windows, the Claude family

2. What is Claude Code - Installation, authentication, commands, IDE setup

3. Capabilities - The agentic loop, multimodal input, headless automation

4. Hands-on: Stock Analysis - Build a real project step by step

5. More Examples - 9 practical tasks: scrapers, APIs, bots, testing, docs

6. Code Quality - Model benchmarks, quality checklists, when not to trust AI

7. Agent Army - Multi-agent orchestration, Agent Teams, git worktrees, tmux

8. MCP & Extensions - Connect to GitHub, Sentry, PostgreSQL, Slack via MCP

9. Best Practices - Context management, prompting strategies, scaling patterns

10. Other Models - OpenRouter, Ollama, LiteLLM, Cline & Kilo Code

The course is free on GitHub: github.com/surfalytics/ClaudeCode101

Each module is a self-contained markdown file with code examples, tables, and practical exercises. Start from Module 01 or jump to whatever interests you most.

Want to go deeper?

Here are some other great (also free) resources:

• Claude Code: A Highly Agentic Coding Assistant — DeepLearning.AI + Anthropic, ~2 hours

• Claude Code in Action — Official Anthropic course

• Claude Code on Coursera — Vanderbilt University, ~5 hours

• Claude Code for Everyone — Community course

Getting started

If you want to try Claude Code right now:

# 1. Install

curl -fsSL https://claude.ai/install.sh | bash

# 2. Navigate to any project

cd your-project

# 3. Start Claude Code

claude

# 4. Ask it anything

> Explain what this project does and suggest improvements.

You’ll need either an Anthropic API key or a Claude Pro/Max subscription. The Pro plan starts at $20/month.

The future of data work isn’t about writing every line by hand — it’s about knowing how to direct AI agents effectively. The sooner you start building that muscle, the better.

You can use Claude Code in CLI or in IDE such as VS Code. If you are new, use with VSCode, if you are pro, feel free to run CLI only.

Connect with me on LinkedIn or visit surfalytics.com for more data engineering content.

We are entering a new era. Not just AI assistants that answer questions. Not just copilots that suggest code. We are talking about teams of AI agents that plan, split work, and build things in parallel.

Think about it. A year ago, we had one AI chat window. Today, we have agent factories.

Here is what the landscape looks like right now:

The pattern is clear. Big vendors are not just building better chatbots. They are building factories of agents. One agent plans the work. Others execute. Some review. All of them talk to each other.

And the open-source community is catching up. Tools like Multi Claude and Gas Town let you run your own fleet of agents right now.

This is not the future. This is February 2026.

I decided to test this with a real project. Not a toy. Not “build me a todo app.” I wanted to see what happens when you give agent teams a real Data Engineering infrastructure task.

The Idea: Build an Azure Data Warehouse With AI Agents

I run a project called Surfalytics where I teach data engineering. I just finished a 2-hour session about Azure reference architectures and building solutions with and without agents. The recording didn’t work out, so I decided to do the exercise myself and share the results.

The idea was simple. Build a complete Azure Data Warehouse environment. 100% delegated to Claude Code Agent Teams.

My role: give the prompt and watch.

Here is what I asked for:

I want to use the team to build a kind of Azure Data Warehouse using:

• Azure SQL Server as my source database

• Azure CosmosDB as my source NoSQL database

• Azure Postgres as my data warehouse

• Azure Data Factory to load data from sources to Postgres (destination)

• Azure DevOps Repos for the code

You can start from a new resource group rg-surfalytics-ai-agents

and create all resources. Make sure they are in one Region.

You can generate sample data in SQL Server (OLTP) and Documents for CosmosDB.

We did az (Azure CLI) and it is logged into the Azure Subscription.

You can use team of agents to split the work and make sure we have a working solution in Azure.

That’s it. A napkin-level description. No detailed spec. No architecture diagram. No step-by-step plan.

Let’s see what happened.

Azure Reference Architecture

Before spending millions of tokens, it is worth checking Azure Reference Architectures to get an idea of typical analytics solution design on Azure

Obviously, it is still a standard System Design framework, but full of Azure icons. Typical use cases are:

• Data Warehouse

• Streaming Analytics

• Lakehouse

• BI Dashboards

• ML solutions

• Generative Apps

• etc.

Obviously, AI can write code for you, but it is not good at judgment, and that’s why the most important skill now is System Design. You can’t fake it, so invest your time.

In addition, you can check our self-paced projects in the Surfalytics repo https://github.com/surfalytics/data-projects

How It Went: Agents at Work

Team Creation and Task Planning

Claude Code created a team of 3 agents and broke the work into 6 tasks with dependencies:

#1 Infrastructure ──→ #2 SQL Data    ──┐
                  ──→ #3 CosmosDB Data ──→ #5 ADF Pipelines ──→ #6 DevOps + CI/CD
                  ──→ #4 DWH Schema  ──┘

The lead agent coordinated everything. When the infra-agent finished task #1, the lead reassigned it to help with task #4 (warehouse schema) and later task #3 (CosmosDB data). Smart delegation.

What the Agents Actually Built

The whole session took 4 hours 9 minutes of agent work time.

Here is what I got in Claude Code:

The agents used only Azure CLI for everything. No Terraform. No ARM templates. No Portal clicks. Just az commands. And they didn’t need any MCP server or special plugins.

Agents even started pipelines to validate the work:

I evaluated the work on a scale from 1 to 5, where 1 is very bad:

Now, we can go into details.

Where Things Broke: The Honest Review

Problem 1: Pipelines Didn’t Work

When everything was finished, I opened Azure Data Factory. The pipelines were there. But they didn’t run.

The datasets were created as empty shells with no linked service references. The pipeline error was clear:

The linked service referenced by the source data set is not found in the copy activity

I asked the agents to fix it. They did. It took several rounds because of:

• Column name mismatches between the source and the warehouse

• Missing source columns referenced in queries

• Surrogate key alignment issues

• Sequence reset problems after TRUNCATE

8 separate fixes before the master pipeline ran green:

Master Pipeline — Succeeded (158s)
┌─────────────────────────────────┬───────────┬──────────────┐
│            Activity             │ Rows Read │ Rows Written │
├─────────────────────────────────┼───────────┼──────────────┤
│ CopyCustomersToDimCustomers     │ 120       │ 120          │
│ CopyProductsToDimProducts       │ 60        │ 60           │
│ CopyOrdersToFactOrders          │ 1,668     │ 1,668        │
│ CopyProductReviewsToFactReviews │ 250       │ 250          │
│ CopyWebEventsToFactWebEvents    │ 600       │ 600          │
└─────────────────────────────────┴───────────┴──────────────┘
Total: 2,698 rows loaded

Lesson: I didn’t mention testing in my prompt. The agents built things but didn’t validate that the end-to-end flow actually works. Always define success criteria upfront.

Problem 2: Passwords in Plain Text

All passwords ended up right in the pipeline code. Connection strings with plain text credentials.

I asked the agents to use Azure Key Vault. They created the Key Vault and stored all secrets. But they didn’t update the ADF linked services to read from Key Vault.

Component: Azure Key Vault
Details: surfalytics-kv — 4 secrets stored
But: ADF still used hardcoded passwords

Lesson: My fault. I didn’t specify security requirements. The agents did exactly what I asked. Nothing more.

Problem 3: Dimensional Modelling Was Bad

This was the worst part.

CosmosDB had 3 collections. SQL Server had 4 tables. I expected some kind of dimensional model. What I got: 3 pipelines with simple COPY activities. No real transformations. No business logic. No proper fact-dimension relationships.

Even after I asked for watermark tables, incremental loading, and SCD Type 2, the results were weak:

• The agents added logging pipelines (not how I would do it)

• For SCD they created INSERT/UPDATE blocks but labeled the INSERT as MERGE

• The data flow DSL was created but the actual transformation logic was simplistic

Score for Dimensional Modelling: 1 out of 5. Even with hints and corrections, the agents couldn’t deliver what a data engineer with DWH experience would build.

What I Added After the Initial Build

After the first round, I asked for three more things:

1. Watermark Table for Pipeline Logging

The agents created a pipeline_watermark table that tracks every execution:

CREATE TABLE IF NOT EXISTS warehouse.pipeline_watermark (
    watermark_id        SERIAL          PRIMARY KEY,
    pipeline_name       VARCHAR(200)    NOT NULL,
    activity_name       VARCHAR(200),
    source_system       VARCHAR(100),
    start_time          TIMESTAMP,
    end_time            TIMESTAMP,
    rows_read           BIGINT,
    rows_written        BIGINT,
    status              VARCHAR(50)
);

After a full pipeline run, the watermark table showed:

ID  Pipeline               Activity          Rows   Status
9   sql-to-postgres        CopyCustomers     120    Succeeded
10  sql-to-postgres        CopyProducts      60     Succeeded
11  sql-to-postgres        CopyOrders        1,668  Succeeded
13  cosmosdb-to-postgres   CopyReviews       250    Succeeded
14  cosmosdb-to-postgres   CopyWebEvents     600    Succeeded

This took some back-and-forth. The first version had SQL syntax errors in the logging queries. But after a few fixes, it worked.

2. SCD Type 2 Data Flow

The agents created a Data Flow for Slowly Changing Dimensions on the dim_customers table. The idea was right:

• Stage incoming records

• Compare with existing dimension using MD5 hash

• Expire changed records (set is_current = false)

• Insert new versions

But the actual implementation was basic. A real data engineer would design this differently.

3. Daily Trigger

A schedule trigger running the master pipeline at 6:00 AM UTC every day. This one worked perfectly out of the box.

Trigger: DailyWarehouseRefresh
Status: Started
Schedule: Every day at 6:00 AM UTC
Pipeline: master-pipeline

Lessons Learned

1. Prompt Engineering Is Everything

Look at my original prompt again. It was a napkin sketch. And that’s exactly what I got back: a napkin-level solution.

The infrastructure part was great because infrastructure is well-defined. “Create a SQL Server” has a clear outcome. But “build a dimensional model” is vague. The agents filled in the blanks with the simplest possible approach.

What I should have done: Take my napkin prompt, feed it to an LLM, and ask for a detailed spec with:

• Exact table schemas for the warehouse

• Transformation logic for each pipeline

• Data quality rules

• Testing criteria

If I had spent 30 minutes on a proper spec, the result would have been completely different.

2. Always Define Success Criteria

I never said “make sure the pipelines actually run.” I never said “validate data lands in the warehouse.” I never said “test the end-to-end flow.”

The agents treated “create a pipeline” as success. They didn’t test if it works.

Rule: Every task needs acceptance criteria. “Create ADF pipeline” is not enough. “Create ADF pipeline that copies 120 customers from SQL Server to dim_customers in PostgreSQL, verify row counts match” is what you need.

3. Security Is Not Automatic

AI agents will take the shortest path. If you don’t mention Key Vault, they will hardcode passwords. If you don’t mention RBAC, they will use admin accounts. If you don’t mention encryption, they won’t enable it.

Rule: Always include security requirements in your prompt. Treat it as a first-class requirement, not an afterthought.

4. Infrastructure: Excellent. Business Logic: Weak.

The pattern was clear:

Agents are great at well-defined, documented tasks. They struggle with tasks that need domain knowledge and design thinking.

5. YOLO Mode Would Save Time

I didn’t use --dangerously-skip-permissions mode. Every az command needed approval. I pressed “accept” about 100 times during the session.

For a controlled environment like this (pet project, no production data), YOLO mode would have saved a lot of time.

What I Would Do Differently

Before the Session

1. Write a detailed spec with exact schemas, transformations, and acceptance criteria

2. Define security requirements upfront (Key Vault, managed identities, no plain text secrets)

3. Include testing instructions (”after creating pipelines, run them and verify row counts”)

4. Use YOLO mode for pet projects to avoid 100+ manual approvals

During the Session

1. Check intermediate results instead of waiting for everything to finish

2. Be specific about data modelling — provide the star schema design yourself

3. Ask agents to validate their own work before reporting “done”

The Spec I Would Write Today

Build an Azure Data Warehouse with these exact specifications:

## Infrastructure
- Resource Group: rg-surfalytics-ai-agents (East US)
- Azure SQL Server with AdventureWorks-style OLTP schema
- Azure CosmosDB with UserProfiles, ProductReviews, WebEvents
- Azure PostgreSQL as data warehouse
- Azure Data Factory for orchestration
- Azure Key Vault for ALL credentials (no plain text anywhere)

## Data Warehouse Schema (PostgreSQL)
- dim_customers (customer_key, customer_id, name, email, 
                 loyalty_tier, is_current, effective_from, effective_to)
- dim_products (product_key, product_id, name, category, price)
- dim_dates (date_key INT YYYYMMDD, full_date, year, quarter, month)
- fact_orders (order_key, customer_key, product_key, 
              order_date_key, quantity, amount)
- fact_reviews (review_key, customer_key, product_key, 
               review_date_key, rating, review_text)
- fact_web_events (event_key, customer_key, event_date_key, 
                   event_type, page_url)

## Pipelines
- sql-to-postgres: Load dims first, then facts with proper key lookups
- cosmosdb-to-postgres: Load reviews and web events with key lookups
- master-pipeline: Orchestrate with logging to pipeline_watermark table
- SCD Type 2 for dim_customers using ADF Data Flow

## Testing
- After creating each pipeline, RUN it and verify row counts
- After full load, query warehouse and confirm data integrity
- Test the daily trigger fires correctly

## Security
- All credentials in Azure Key Vault
- ADF uses managed identity to access Key Vault
- No passwords in code, config files, or pipeline definitions

This spec would give the agents clear direction. No ambiguity. Clear success criteria.

The Cost

I ran Claude Code with Opus 4.6 model on the Anthropic API. No subscription limits. Pure token-based billing.

The $40 also included some work tasks I ran that evening, so the actual cost for this exercise was probably around $25-30.

For context: setting up the same environment manually would take me 2-3 hours. The agents did it in about 4 hours with my supervision. But I was mostly watching, not working. The effort on my side was maybe 30 minutes of actual input.

If I had a better spec, the agent time would likely drop to 2 hours with fewer fix cycles.

The Risk for New Engineers

Here is the part I want you to read carefully.

If you are a junior engineer, you are in danger.

Not because AI will replace you. Because AI makes it too easy to get a “good” result without understanding what happened.

The agents stored passwords in plain text. They created pipelines that looked right but didn’t work. They built a “dimensional model” that was just COPY activities. They labeled an INSERT block as MERGE.

If you have experience, you catch these things immediately. I saw the hardcoded passwords and said “put them in Key Vault.” I saw the broken pipelines and knew what to fix. I looked at the “dimensional model” and knew it was wrong.

But a junior engineer? They would see:

• Green checkmarks on all tasks

• Resources created in Azure

• Pipelines in ADF

• A “data warehouse” with tables

And they would think: “It works. I’m done.”

They would deploy this to production. They would not know about:

• Security best practices

• Proper dimensional modelling

• Data quality testing

• Incremental loading patterns

• Error handling

You can work for months with AI doing your job. Your resume will show years of experience. But those years won’t count as real experience. You won’t understand the fundamentals. You won’t catch the mistakes.

Why Am I Effective With AI?

It’s not because AI is amazing (it is). It’s because I have 15 years of experience. The first 12-13 years, I did everything by hand. I googled every error. I debugged every pipeline. I learned why things break.

Now AI helps me do faster what I already know how to do. And that’s the key difference. I can review the AI’s output because I know what “good” looks like.

So be careful with AI. Use it as a tool. But learn the fundamentals first. Understand what the AI builds for you. Don’t just accept green checkmarks.

What You Can Do With This Approach

This exercise shows that agent teams can build real infrastructure. Here are some ideas:

• Azure / AWS / GCP learning environments — Let agents build the infrastructure, then study and modify it

• Pet projects for your portfolio — Get a working stack quickly, then improve it manually

• Testing new architectures — Want to try a lakehouse? Give agents the spec and see what happens

• Batch and streaming pipelines — Create both patterns and compare

• Data modelling exercises — Build the warehouse schema yourself, let agents handle the infrastructure

My Next Exercise

I want to do the same thing but with open-source tools. Everything running on local Docker or Kubernetes. PostgreSQL instead of Azure SQL. MinIO instead of Azure Blob. Apache Airflow instead of ADF.

But this time, I will feed the agents a proper spec. Let’s see the difference.

Key Takeaways

1. Agent teams are real. Claude Code, Cursor, and others can coordinate multiple agents on complex tasks. This is not a demo. This works today.

2. Infrastructure tasks are a sweet spot. Agents excel at creating cloud resources, setting up CI/CD, and generating sample data.

3. Business logic is still hard for agents. Dimensional modelling, complex transformations, and domain-specific design need human guidance.

4. Your prompt is your spec. Napkin sketch in, napkin solution out. Detailed spec in, production-quality solution out.

5. Always define success criteria. “Create a pipeline” is not enough. “Create a pipeline, run it, verify 120 rows land in dim_customers” is what works.

6. Security is your responsibility. Agents won’t add security unless you ask for it.

7. The cost is reasonable. $25-30 for a complete Azure environment setup. $2/day for running resources.

8. For experienced engineers: golden time. AI amplifies what you already know. Use it.

9. For new engineers: learn the basics first. Don’t let AI skip your learning. Understand what you’re building before you delegate.

PS: I genuinely enjoyed this process. In some way, sessions with Claude Code replaced my need for doom scrolling. Building things with AI agents is more fun than social media.

PPS: The entire Azure environment is in one resource group. When you’re done exploring, one command cleans everything up:

az group delete --name rg-surfalytics-ai-agents --yes --no-wait

1. AI avoiders: Those who rely only on traditional development practices and ignore AI tools entirely

2. Over-reliant users: Those who copy-paste everything into ChatGPT for all their tasks

3. AI enthusiasts: Those too deep in AI development—constantly experimenting with the latest models and techniques

The optimal approach lies between groups 2 and 3. Developers/Engineers and analysts should:

• Understand AI capabilities and limitations

• Apply AI strategically to appropriate tasks

• Stay informed without getting distracted by every new release

This balanced approach maximizes both speed and quality gains.

In this blogpost, I want to highlight the available list of tools and cover some of the definitions.

TL;DR:

• Get Cursor IDE.

• Use Auto mode.

• Add rules per repo.

• Add MCP integration for tools you are using.

Don’t worry if something unclear, just use it daily and you will be fine.

1. Key Terms (Before Diving into AI Dev Assistants)

Understanding these terms will help you choose the right tools, control costs, and get better results.

1.1 LLM (Large Language Model)

An LLM is a model trained on huge amounts of text (and sometimes code) that can generate text, answer questions, and follow instructions. When you chat with an AI assistant or ask it to write code, you are talking to an LLM.

Simple example: You type “Write a SQL query to get the last 7 days of orders.” The model reads your text and generates SQL. It does not run the query; it only produces the text.

1.2 LLM Model Providers and Models

A model provider is the company or service that hosts and serves the models. You send your prompt to their API and get a response.

Models from the same provider can differ in:

• Speed – faster models are usually cheaper and sometimes less capable.

• Context window – how much text (and code) they can read at once (e.g. 32K, 200K tokens).

• Reasoning vs non-reasoning – see below.

1.3 Reasoning vs Non-Reasoning Models

Non-reasoning (standard) models
They read your prompt and generate a response in one pass. They are fast and cost less per request. Good for: short answers, simple code, edits, and most day-to-day tasks.

Reasoning models (e.g. OpenAI o1, o3; some “thinking” modes)
They spend extra “thinking” steps before answering. They can be better for hard logic, math, or multi-step planning, but they are slower and use more tokens (so they cost more). Use them when you need deeper reasoning, not for every request.

Practical tip: For most data work (SQL, dbt, Python, docs), a good non-reasoning model (e.g. Claude 3.5 Sonnet, GPT-4o) is enough. Switch to a reasoning model only when you hit a genuinely hard problem.

1.4 Tokens

Tokens are the units the model uses to read and write text. Roughly:

• 1 token ≈ 4 characters in English, or about ¾ of a word.

• Code and symbols often use more tokens per character.

Example of model cost from Antropic API:

Why it matters:
Providers charge by token (input and sometimes output). Long prompts, long replies, and big context windows increase token usage and cost. Team or enterprise plans often have token quotas or dollar-based limits.

Simple example:

• Prompt: “Write a SELECT that returns the top 10 customers by revenue.” → on the order of 10–20 tokens.

• Response: a 20-line SQL query → maybe 100–200 tokens.

• If you paste a 500-line dbt model into the prompt, that can be thousands of tokens per request.

1.5 Embedding and Indexing

Embedding
An embedding is a numerical representation of a piece of text (or code). Similar meaning → similar numbers. Models use embeddings to search and compare text.

Indexing (in the context of AI IDEs)
Indexing means the tool (e.g. Cursor) builds a searchable “map” of your repo or docs using embeddings. When you ask a question, the tool finds the most relevant files or snippets and sends them to the model as context. So the model can “see” your codebase or documentation without you pasting everything manually.

Simple example:
You have a dbt project and external API docs. You index the repo and the docs. You then ask: “Create a bronze dbt model for the Amplitude events table using our project style.” The tool finds the right source file, existing models, and relevant docs, and the model generates the new model using that context.

1.6 MCP (Model Context Protocol)

MCP is an open protocol that lets an AI assistant (in an IDE or CLI) call external tools on your behalf. For example:

• Query a database (e.g. list schemas, get DDL, run a SELECT).

• Call cloud APIs (e.g. AWS, Snowflake).

• Read from files, run CLI commands (when allowed by the tool).

So instead of you running the query and pasting the result, you tell the assistant “list tables in schema X” or “get the DDL for table Y”; it uses MCP to run the right command and use the result in the answer.

Or similar MCP for Claude Code in JSON file .mcp.json:

{
  "mcpServers": {
    "snowflake_solo": {
      "command": "/Users/github/dbt/.venv/bin/python3",
      "args": [
        "/Users/github/dbt/start_solo_mcp_simple.py"
      ]
    }
  }
}

Why it matters for data work:
You can ask the AI to inspect Redshift/Snowflake (schemas, tables, DDL), pull CloudWatch logs, or list AWS resources. It can then generate SQL, dbt, or docs using live metadata. Caution: MCP can use many tokens (lots of context and tool results), so watch usage and cost when you enable it.

1.7 Rules for Agents (Why They Matter and What to Put in Them)

Rules (or “agent rules”, “project rules”) are instructions that the IDE or agent applies to every (or chosen) conversations. They steer style, conventions, and safety without you repeating the same instructions in each prompt.

Why they are important:

• Consistency – same SQL style (e.g. lowercase keywords), same dbt structure, same doc format.

• Quality – enforce lint rules, testing, or “always add a short comment for complex logic.”

• Efficiency – you don’t re-explain your stack or standards every time.

What you can put in rules (examples for data engineers/analysts):

• SQL: lowercase keywords, table aliases, CTE naming, no SELECT * in production models.

• dbt: layer naming (e.g. bronze/silver/gold), source and model naming, always add schema.yml and tests where needed.

• Python: style (e.g. Black), type hints for public functions, docstrings for pipelines.

• Docs: format (e.g. Markdown), what to document (sources, columns, refresh logic).

• Stack: “We use Airflow for orchestration, dbt for transformations, Snowflake/Redshift as the warehouse.”

• Security: “Never put credentials in code”; “Use key-pair auth for Snowflake when possible.”

You can have global rules (all projects) and project-specific rules (e.g. only for the “analytics-dbt” repo). In Cursor these live in .cursor/rules; in Claude Code you can use an AGENTS.md or similar file and reference it in the project.

2. Use Cases for Data Engineers and Data Analysts

These are concrete ways teams use AI assistants in data work.

Important: Treat the AI as a junior assistant. You should understand the problem, the design, and the data; use the AI to speed up implementation, drafts, and troubleshooting, not to replace your judgment.

3. Tools Overview

From the simplest and most convenient to the more configurable or organization-managed options.

3.1 Cursor

What it is:
An AI-first IDE built on VS Code. It focuses on ease of use: rules, codebase indexing, and MCP are integrated in one place.

Why it’s convenient:

• Rules – Project and global rules so the model follows your SQL/dbt/Python conventions without repeating them.

• Indexing – Index repos and external docs (e.g. API docs via URL); the model uses this for context automatically.

• MCP – Configure MCP servers (e.g. Snowflake, Redshift, AWS) once; use them from chat or Composer.

• Models – You can use several providers; many users find Anthropic models (e.g. Claude 3.5 Sonnet/Opus) very good for coding.

• Composer and Auto – Cursor’s own “Composer” and Auto mode are optimized for multi-file edits and long tasks; they often feel generous on quotas and can feel “unlimited” for typical use compared to strict per-request limits elsewhere.

Pricing (highlight):

• Personal: Paid plan around $20/month (e.g. $20 for Pro); often includes a set of “fast” requests and then usage-based.

• Teams / Business: Per-seat; higher limits and team management.

• Check cursor.com for current tiers and token/request limits. Composer and Auto may have different quota rules than standard chat.

3.2 Claude Code (Anthropic)

What it is:
Anthropic’s coding agent. You can use it from the CLI (terminal) or via the Claude extension in VS Code. It’s the same “Claude” chat experience, with the ability to run commands, read/write files, and use MCP when configured.

How people use it:

• CLI – Run Claude (or similar) in a terminal; chat and give tasks in a project folder. No IDE required.

  

• VS Code – Install the Claude Code / Anthropic extension; get a chat panel and agent behavior inside VS Code.

  

Benefits:

• Strong models (e.g. Claude 3.5 Sonnet/Opus, Claude 4.5) that many developers like for code.

• Same account and behavior in CLI and VS Code.

• MCP and custom instructions (like rules) are supported.

Models:
Claude Code uses Anthropic models only (it does not switch to OpenAI or others). You can choose model size in settings (e.g. Sonnet vs Opus).

Tokens and pricing:

• Usage is token-based; you have daily/monthly quotas depending on plan.

• Teams (and Enterprise) have dollar-based or higher token quotas – it’s easy to hit limits if many people use it heavily (e.g. PR reviews, large context).

• Tip: Monitor usage; there are known cases of teams burning through large monthly quotas (e.g. in CI or with MCP), so set alerts if your org uses Teams.

3.3 Third-Party Plugins: Kilo Code and Cline (VS Code)

What they are:
Extensions for VS Code that give you an AI chat/agent experience and can use multiple model providers (e.g. OpenAI, Anthropic, Open Router).

Kilo Code (e.g. from kilo.ai):

• Connects to your API keys (Anthropic, OpenAI, or Open Router).

• Open Router acts as a gateway to many models (Claude, GPT, Gemini, DeepSeek, etc.) with one key.

• Good when Claude Code is blocked or you want to switch providers without changing IDEs.

• Shows cost per request (e.g. “$0.74 for this request”), which helps with token awareness.

Cline (formerly Claude Dev):

• Another VS Code agent that can use different backends (e.g. OpenAI, Anthropic).

• Similar idea: chat, code generation, sometimes MCP; you bring your own API key.

Benefits:

• Stay in VS Code with your existing extensions.

• Any supported provider – not tied to one vendor.

• Open Router (with Kilo or similar) – one API key, many models; useful for comparing models or using what your org allows.

3.4 AWS Bedrock

What it is:
AWS’s managed service for foundation models. You don’t talk to OpenAI or Anthropic directly; you call Bedrock APIs and choose from the models AWS offers (including some from Anthropic, Meta, Mistral, etc.).

Why it matters for teams on AWS:

• Data and compliance – Data stays in your AWS account and region; useful for data governance.

• Unified billing – Model usage on one AWS bill.

• Enterprise controls – IAM, VPC, logging.

How it can work with VS Code:

• Some extensions or custom integrations allow “Bedrock” as a model backend (e.g. via API endpoint and AWS auth). You may need to check the marketplace or your internal tools.

• Alternatively, you use Bedrock from your own scripts or internal tools; the “IDE” part might still be Cursor, Claude Code, or a VS Code plugin that supports a Bedrock-compatible API.

Models:
Bedrock offers a subset of models (e.g. Claude, Llama, Mistral, others). Not every model from every provider is available; check the current Bedrock model list for your region.

3.5 Azure (Azure OpenAI / Azure AI Studio)

What it is:
Microsoft’s managed service for foundation models in Azure. Similar in role to AWS Bedrock: you call Azure APIs instead of talking to OpenAI or other providers directly. Azure OpenAI Service offers OpenAI models (GPT-4o, etc.) in your tenant; Azure AI Studio provides a single place to use Azure OpenAI and other models (including some open and third-party models).

Why it matters for teams on Azure:

• Data and compliance – Data stays in your Azure tenant and region; fits Microsoft-centric governance and enterprise agreements.

• Unified billing – Model usage on your Azure bill.

• Enterprise controls – Azure AD, RBAC, private endpoints, audit logs.

How it can work with VS Code:

• GitHub Copilot (when backed by your org’s subscription) can use Azure OpenAI where configured.

• Extensions or internal tools can call Azure OpenAI (or Azure AI) APIs with Azure auth; you use your usual IDE with a Bedrock-like “bring your own cloud” backend.

Models:
Azure OpenAI offers OpenAI models (e.g. GPT-4o, GPT-4o mini, o1). Azure AI Studio may offer additional models; check the Azure AI Studio and Azure OpenAI docs for the current list in your region.

3.6 GCP (Vertex AI)

What it is:
Google Cloud’s managed platform for foundation models. Vertex AI lets you use Gemini, and often other models (e.g. open or partner models), via Google’s APIs. Like Bedrock and Azure, your data and usage stay in your GCP project.

Why it matters for teams on GCP:

• Data and compliance – Data stays in your GCP project and region; useful for Google-centric data governance.

• Unified billing – Model usage on your GCP bill.

• Enterprise controls – IAM, VPC-SC, audit logging.

How it can work with VS Code:

• Some extensions or internal tools support Vertex AI as a model backend (e.g. via REST or SDK with GCP auth). You code in VS Code while the model runs in your project.

• Alternatively, you call Vertex AI from scripts or pipelines; the “IDE” part stays in Cursor, Claude Code, or a plugin that can target a compatible API.

Models:
Vertex AI offers Gemini (e.g. Gemini 1.5 Pro/Flash, Gemini 2.0) and often other models. Check the Vertex AI model garden for the current list and regions.

3.7 Google Antigravity (IDE alternative)

What it is:
Google Antigravity is Google’s AI-first IDE and agentic development platform (launched in preview in 2025). Unlike “AI inside VS Code,” it is a dedicated environment where AI agents can plan work, edit multiple files, run and debug code, and be monitored from an Editor view and a Manager surface. It is positioned as an alternative to Cursor or Claude Code for teams open to a Google-native, Gemini-based workflow.

Why consider it:

• Free in preview – Currently available at no charge for personal Gmail accounts (preview).

• Gemini-powered – Uses Google’s latest models (e.g. Gemini 3 Pro) tuned for code and multi-step tasks.

• Agent-centric – Task-based agents, artifacts, verification, and optional multi-agent orchestration.

• Two modes – Editor view for hands-on coding with inline AI; Manager surface to spawn and observe agents across workspaces.

Relevance for data engineers/analysts:

• Same use cases as other AI IDEs: SQL, dbt, Python, docs, refactors. Quality will depend on Gemini’s code and reasoning support and how well your stack (e.g. dbt, Airflow, cloud) is represented in context.

• Good to evaluate if your org is already on GCP or considering Gemini; can complement or substitute Cursor/Claude Code for day-to-day coding.

Caveats:

• Preview product; roadmap and enterprise/team plans may change. Check antigravity.google and Google’s developer docs for current availability and terms.

3.8 GitHub Copilot

What it is:
Microsoft’s AI pair programmer: inline completions, chat, and (in Copilot+) more agent-like features. It integrates with VS Code (and other editors) and is widely used in organizations that already have Microsoft/GitHub agreements.

Why include it:

• Many organizations already have Copilot (per-seat or as part of a deal). It’s the default “AI in the editor” for a lot of developers.

• Tight VS Code integration – Completions as you type, chat in the sidebar, and (where available) agent features.

Relevance for data engineers/analysts:

• Good for inline code (SQL, dbt, Python, YAML for Airflow/Terraform) and short explanations.

• Less emphasis (today) on deep “index whole repo + MCP + rules” in the way Cursor or Claude Code do; often used as a daily driver for suggestions and quick chat, with other tools for heavy lifting.

4. Summary Table

Summary

As I mentioned in the beginning, you need to have an IDE that allows you to leverage the power of one of the top LLM models from OpenAI, Gemini, or Antropic and just start to use them in your daily routine.

If you want to see a real demo of how we are using AI, I have a session for you:

You want to learn data engineering. But where do you start?

Courses teach theory. YouTube videos show demos. But you need hands-on experience. You need to build something real.

Here’s the problem. Cloud accounts need credit cards. Tools cost money. And most tutorial projects feel fake. They don’t match what companies actually do.

What if you could build a real data engineering project for free? Using the same tools that companies use? Without a credit card?

That’s what this guide will help you do.

What You Will Build

By the end of this guide, you will have:

• A working data pipeline that runs every day

• Real data from GitHub in your data warehouse

• Transformed data ready for dashboards

• A BI dashboard showing insights

• A portfolio project you can show and tell in interviews

This is not a toy project. This is how many companies build their data solutions. The tools we use (Fivetran, Snowflake, dbt) are industry standards. You will learn patterns used in production systems.

Let’s get started.

Part 1: The Architecture

Before we build, let’s design. Good data engineers think about the big picture first.

The System Design Framework

Every data solution has the same basic parts:

1. Source - Where data comes from

2. Ingestion - How data moves to storage

3. Storage - Where data lives

4. Transformation - How data gets cleaned and shaped

5. Consumption - How people use the data

I’ve covered this in detail in another blog post:

For our project, we will use the same approach as we would like to use (and know) for any data analytics project:

Why These Tools?

Source: GitHub API

• Free to use

• Rich data available

• No business account needed

Ingestion: Fivetran

• No code required

• Reliable

• 14-day free trial

Storage: Snowflake

• Most popular data warehouse

• $400 in free credits

Transformation: dbt Cloud

• Industry standard tool

• Free developer plan

BI: Looker Studio

• Free forever

• Easy to learn

Part 2: The Data Source - GitHub API

What Data Will We Get?

GitHub stores data about code projects. The GitHub API lets us access this data. Fivetran has a connector that pulls this data for us.

We will get tables like:

• Repositories - Project name, description, language, stars, forks

• Commits - Code changes, authors, dates

• Issues - Bug reports, feature requests

• Pull Requests - Code review requests

• Users - People who contribute

What Questions Can We Answer?

With this data, we can build dashboards that answer:

1. Which programming languages are most popular?

2. Which repos have the most stars?

3. How fast are repos growing?

4. Who are the most active contributors?

5. How many issues get closed vs stay open?

These are real business questions. Companies ask similar questions about their own data.

Alternative Data Sources

Don’t want GitHub data? You can use other free sources:

• Relational database such as Postgres (Neon, RDS), SQL Server

• Cloud Storage (S3, Azure Blob, Google Storage)

• NoSQL database (MongoDB)

• Numerous other APIs

What is an API?

An API (Application Programming Interface) is a messenger that lets different software applications talk to each other. Think of it as a waiter who takes your order to the kitchen and brings back your food.

Just think about it as a way to get the data from the external application.

How It Works

1. You make a request → Ask for data

2. API processes it → Finds what you need

3. You get a response → Receives the data

Basic Components

Endpoint: The URL where you access the API

https://api.example.com/users

Request Methods:

• GET - Retrieve data

• POST - Create new data

• PUT - Update data

• DELETE - Remove data

Response: Data returned (usually JSON format)

{

“name”: “John”,

“age”: 30

}

Part 3: Ingestion with Fivetran

What is Fivetran?

Fivetran is a data ingestion tool. It connects to data sources and copies data to your data warehouse. You don’t write code. You just click buttons.

Fivetran handles the hard parts:

• Connecting to APIs

• Handling errors and retries

• Tracking what changed (incremental loading)

• Keeping schemas in sync

How Often Should Data Sync?

This is an important decision. If you sync more often, it costs more money. If you sync less often, your data may be old.

Common sync options:

• Every 15 minutes. Because Fivetran can.

• Every hour: Best for business-critical dashboards.

• Every 6 hours: Good for daily operations. Medium cost.

• Every 24 hours: For historical analysis. Lowest cost.

For our project, we will sync once per day. This works well with the free trial. Daily syncs are usually enough for analytics dashboards.

The biggest disadvantage for Fivetran is its pricing model. Unlike traditional per-user or flat-rate subscriptions, Fivetran charges based on MAR (Monthly Active Rows), which can quickly become unpredictable and expensive as your data scales.

What is MAR?

Monthly Active Rows counts every row of data that Fivetran modifies, creates, or updates during a billing period. This includes:

• New rows added to your tables

• Existing rows that get updated

• Rows that are re-synced or backfilled

Why MAR Pricing is Problematic

• Unpredictable Costs: If your data changes frequently, your bill can spike unexpectedly. A single table with daily updates can generate millions of MARs.

• Penalizes Data Growth: As your business grows and generates more data, your Fivetran costs grow exponentially—not linearly.

• Hidden Complexity: Calculating MAR isn’t straightforward. Different connectors count rows differently, making it hard to forecast expenses.

• Backfill Nightmares: Need to re-sync historical data? Every row counts toward your MAR, potentially doubling or tripling your monthly bill.

• No Cost Control: Unlike compute-based pricing where you can pause resources, with MAR you’re charged for data movement whether you’re actively using it or not.

The Bottom Line

While Fivetran offers excellent reliability and ease of use, their MAR pricing model can lead to budget overruns and makes it difficult for growing companies to predict their data infrastructure costs. Many teams find themselves paying thousands more than anticipated once their data volume scales.

Alternatives to Fivetran

Fivetran is not the only tool you can use. Here are some other choices:

• Fivetran: Managed and no-code. Good if you want to start quickly and don’t want to code.

• Airbyte: Open source. Choose this if you want more control over the process.

• Meltano: Open source and uses the command line. Best for developers who like working with CLI tools.

• Custom Python: Write code to handle data syncing. Good for special situations or unique needs. Usually run by Airflow.

For learning, Fivetran is the easiest. Start here. Try alternatives later.

Part 4: Storage with Snowflake

What is Snowflake?

Snowflake is a cloud data warehouse. Think of it as a big database designed for analytics. It’s where all your data lives.

Snowflake is popular for good reasons:

• Easy to use (SQL-based)

• Scales automatically

• Separates storage from compute (you pay for what you use)

• Great documentation

• Used by thousands of companies

Want to learn more? Check my last book - Jumpstart Snowflake: A Step-by-Step Guide to Modern Cloud Analytics

Setting Up Snowflake

Step 1: Create an account at https://signup.snowflake.com/

Step 2: Create the database structure

Open a SQL worksheet in Snowflake. Run these commands:

-- Create databases for each layer
CREATE DATABASE GITHUB_RAW;
CREATE DATABASE GITHUB_STAGING;
CREATE DATABASE GITHUB_ANALYTICS;

-- Create schemas
CREATE SCHEMA GITHUB_RAW.FIVETRAN;
CREATE SCHEMA GITHUB_STAGING.STAGING;
CREATE SCHEMA GITHUB_ANALYTICS.MARTS;

-- Create a warehouse (compute) for running queries
CREATE WAREHOUSE TRANSFORM_WH 
  WAREHOUSE_SIZE = 'X-SMALL'
  AUTO_SUSPEND = 60
  AUTO_RESUME = TRUE;

What did we just create?

• GITHUB_RAW - Where Fivetran puts raw data

• GITHUB_STAGING - Where we clean data

• GITHUB_ANALYTICS - Where we put final tables for dashboards

• TRANSFORM_WH - A small compute instance to run queries

Step 3: Create users for tools

Each tool needs its own login. This is good security practice.

-- Create a user for Fivetran
CREATE ROLE FIVETRAN_ROLE;
CREATE USER FIVETRAN_USER 
  PASSWORD = 'ChooseAStrongPassword123!'
  DEFAULT_ROLE = FIVETRAN_ROLE;

GRANT ROLE FIVETRAN_ROLE TO USER FIVETRAN_USER;
GRANT USAGE ON WAREHOUSE TRANSFORM_WH TO ROLE FIVETRAN_ROLE;
GRANT ALL ON DATABASE GITHUB_RAW TO ROLE FIVETRAN_ROLE;

-- Create a user for dbt
CREATE ROLE DBT_ROLE;
CREATE USER DBT_USER 
  PASSWORD = 'ChooseAnotherStrongPassword456!'
  DEFAULT_ROLE = DBT_ROLE;

GRANT ROLE DBT_ROLE TO USER DBT_USER;
GRANT USAGE ON WAREHOUSE TRANSFORM_WH TO ROLE DBT_ROLE;
GRANT USAGE ON DATABASE GITHUB_RAW TO ROLE DBT_ROLE;
GRANT SELECT ON ALL TABLES IN DATABASE GITHUB_RAW TO ROLE DBT_ROLE;
GRANT SELECT ON FUTURE TABLES IN DATABASE GITHUB_RAW TO ROLE DBT_ROLE;
GRANT ALL ON DATABASE GITHUB_STAGING TO ROLE DBT_ROLE;
GRANT ALL ON DATABASE GITHUB_ANALYTICS TO ROLE DBT_ROLE;

Save these passwords. You will need them later.

Data Layers Explained

We created three databases. This is called a “layered architecture.” Each layer has a job.

RAW Layer (Bronze)

• Exact copy of source data

• Fivetran manages this

• We never change this data

• If something breaks, we can always go back here

STAGING Layer (Silver)

• Cleaned and typed data

• Column names are consistent

• Bad data is filtered out

• dbt manages this

ANALYTICS Layer (Gold)

• Business-ready tables

• Aggregations and metrics

• Ready for dashboards

• dbt manages this

Some people call this “Medallion Architecture” (Bronze, Silver, Gold). Some call it “Raw, Stage, Marts.” The names don’t matter. The concept does.

Alternatives to Snowflake

Snowflake, BigQuery, Redshift, and Synapse (now called Microsoft Fabric) are the most popular warehouse options. It means you can try another platform:

• BigQuery: Good if you use Google Cloud Platform (GCP). GCP gives you $300 in credits.

• Redshift: Ideal for people using Amazon Web Services (AWS). There is an AWS free tier.

• Synapse: Best for Azure users. Azure gives free credits.

For learning the modern data stack, Snowflake is the best choice. Most job postings mention it.

Part 5: Finish Fivetran Setup

Now that Snowflake is ready, let’s finish Fivetran.

Step 1: Add Snowflake as a destination

1. In Fivetran, go to Destinations

2. Click “Add Destination”

3. Choose “Snowflake”

4. Enter your Snowflake details:

   • Account: Your Snowflake account URL

   • User: FIVETRAN_USER

   • Password: The password you created

   • Database: GITHUB_RAW

   • Schema: FIVETRAN

   • Warehouse: TRANSFORM_WH

5. Test the connection

Step 2: Add GitHub connector

1. Go to Connectors

2. Click “Add Connector”

3. Search for “GitHub”

4. Click “Set Up”

5. Authenticate with your GitHub account

6. Select which repos or organizations to sync

7. Choose sync frequency: Daily

8. Start the sync

Fivetran will now pull data from GitHub and put it in Snowflake. The first sync takes a few minutes. After that, it runs daily.

Part 6: Data Modelling Concepts

Before we build dbt models, let’s learn two important concepts. These come up in every data engineering interview.

Dimensional Modelling

Dimensional modelling is how we organize data for analytics. It’s the most common approach for data warehouses.

The idea is simple. We have two types of tables:

Fact Tables - Store events and measurements

• Things that happen: orders, clicks, commits

• Have numbers you can add up: revenue, count, duration

• Named with the prefix fct_

Dimension Tables - Store descriptions

• Things that describe facts: customers, products, dates

• Have text and categories: name, type, status

• Named with prefix dim_

For our GitHub project:

• fct_repo_activity (Fact table): Stores daily counts of commits, pull requests, and issues for each repository.

• dim_repositories (Dimension table): Stores details about each repository, like name, language, owner, and description.

• dim_users (Dimension table): Stores user information, such as username, name, and profile details.

• dim_dates (Dimension table): Stores date-related details, such as the date, month, quarter, and year.

Star Schema

When you draw this, the fact table sits in the middle. Dimension tables connect to it like points of a star.

This design makes queries fast. BI tools love it.

Slowly Changing Dimensions (SCD)

Here’s a problem. What happens when data changes over time?

A GitHub user changes their username. A repo changes owner. How do we handle this?

This is called “Slowly Changing Dimensions” or SCD. There are different approaches:

SCD Type 1 - Overwrite

Just replace the old value with the new one.

Simple. But we lose history. We can’t see what the old username was.

SCD Type 2 - Keep History

Add new rows for changes. Track when each version was valid.

More complex. But we keep the full history. We can see what the username was at any point in time.

Which to choose?

For our project, we’ll start with Type 1. It’s simpler. You can add Type 2 later using dbt snapshots.

Interview Tip: Interviewers love asking about SCD. Be ready to explain Type 1 vs Type 2 and when to use each.

Part 7: Transformation with dbt

What is dbt?

dbt stands for “data build tool.” It transforms data using SQL.

Here’s how it works:

1. You write SQL SELECT statements

2. dbt runs them and creates tables

3. dbt tracks dependencies between tables

4. dbt tests your data for quality

dbt is the industry standard for data transformation. Almost every modern data team uses it.

dbt is the most important part of this project. It ties together data warehouse, data ingestion, business logic, software engineering practices and so on. You definitely need to invest time in this tool and understand it well.

Check their learning page: https://learn.getdbt.com/catalog

You probably need only the fundamentals course.

Setting Up dbt Cloud

Step 1: Create an account at https://www.getdbt.com/signup

Step 2: Create a GitHub repository

1. Go to github.com

2. Create a new repository called “github-analytics-dbt”

3. Make it public (easier for portfolio)

4. Initialize with a README

We covered this in module 0:"

Step 3: Connect dbt to GitHub

1. In dbt Cloud, create a new project

2. Connect to your GitHub repository

3. dbt will sync code automatically

Step 4: Connect dbt to Snowflake

1. Add a Snowflake connection

2. Enter details:

   • Account: Your Snowflake account

   • User: DBT_USER

   • Password: The password you created

   • Database: GITHUB_STAGING (default)

   • Warehouse: TRANSFORM_WH

3. Test the connection

dbt Project Structure

dbt projects have a standard structure:

Creating dbt Models

Let’s build the models. You have the options here. You can use the dbt labs studio and build all models in web browser. Or you can use IDEs uch as VSCode. We covered this as well:

Or you may want to leverage AI in full swing:

If you will use an IDE, you will need to define a connection to Snowflake and dbt Labs. I highly recommend using this. Moreover, you can use the IDE dbt core (open source) instead of dbt Labs.

Now, let’s add the models.

First, define sources

Create models/staging/_staging.yml:

version: 2

sources:
  - name: github_raw
    database: GITHUB_RAW
    schema: FIVETRAN
    tables:
      - name: repository
      - name: commit
      - name: user
      - name: issue
      - name: pull_request

This tells dbt where the raw data lives.

Create staging models

Staging models, clean and rename data. They don’t change the grain (one row per source row).

Create models/staging/stg_repositories.sql:

WITH source AS (
    SELECT * FROM {{ source('github_raw', 'repository') }}
),

cleaned AS (
    SELECT
        -- Primary key
        id AS repository_id,
        
        -- Attributes
        name AS repository_name,
        full_name,
        owner_login AS owner_username,
        description,
        language,
        
        -- Metrics
        stargazers_count AS stars,
        forks_count AS forks,
        open_issues_count AS open_issues,
        
        -- Dates
        created_at,
        updated_at,
        pushed_at
    FROM source
    WHERE id IS NOT NULL
)

SELECT * FROM cleaned

Create models/staging/stg_users.sql:

WITH source AS (
    SELECT * FROM {{ source('github_raw', 'user') }}
),

cleaned AS (
    SELECT
        id AS user_id,
        login AS username,
        name AS display_name,
        email,
        company,
        location,
        bio,
        public_repos AS public_repo_count,
        followers AS follower_count,
        following AS following_count,
        created_at,
        updated_at
    FROM source
    WHERE id IS NOT NULL
)

SELECT * FROM cleaned

Create mart models

Mart models are business-ready. They aggregate and join data. It means we would have only two layers in our data warehouse - Mart (Gold) and Staging (Bronze)

Create models/marts/dim_repositories.sql:

WITH repositories AS (
    SELECT * FROM {{ ref('stg_repositories') }}
)

SELECT
    repository_id,
    repository_name,
    full_name,
    owner_username,
    description,
    language,
    stars,
    forks,
    open_issues,
    created_at AS repo_created_at,
    updated_at AS repo_updated_at,
    pushed_at AS last_push_at
FROM repositories

Create models/marts/dim_users.sql:

WITH users AS (
    SELECT * FROM {{ ref('stg_users') }}
)

SELECT
    user_id,
    username,
    display_name,
    email,
    company,
    location,
    bio,
    public_repo_count,
    follower_count,
    following_count,
    created_at AS user_created_at
FROM users

Create models/marts/fct_daily_stats.sql:

WITH repositories AS (
    SELECT * FROM {{ ref('stg_repositories') }}
),

daily_stats AS (
    SELECT
        DATE_TRUNC('day', updated_at) AS stat_date,
        language,
        COUNT(*) AS repo_count,
        SUM(stars) AS total_stars,
        SUM(forks) AS total_forks,
        ROUND(AVG(stars), 2) AS avg_stars,
        ROUND(AVG(forks), 2) AS avg_forks
    FROM repositories
    WHERE language IS NOT NULL
    GROUP BY 1, 2
)

SELECT * FROM daily_stats

Configure models

version: 2

sources:
  - name: github_raw
    database: GITHUB_RAW
    schema: FIVETRAN
    tables:
      - name: repository
      - name: commit
      - name: user
      - name: issue
      - name: pull_request

models:
  - name: stg_repositories
    description: "Cleaned repository data from GitHub"
    columns:
      - name: repository_id
        description: "Unique identifier for the repository"
        tests:
          - unique
          - not_null
      - name: repository_name
        description: "Name of the repository"
        tests:
          - not_null
      - name: language
        description: "Primary programming language"

  - name: stg_users
    description: "Cleaned user data from GitHub"
    columns:
      - name: user_id
        description: "Unique identifier for the user"
        tests:
          - unique
          - not_null

Create models/marts/_marts.yml:

version: 2

models:
  - name: dim_repositories
    description: "Dimension table for repositories"
    columns:
      - name: repository_id
        description: "Primary key"
        tests:
          - unique
          - not_null

  - name: dim_users
    description: "Dimension table for users"
    columns:
      - name: user_id
        description: "Primary key"
        tests:
          - unique
          - not_null

  - name: fct_daily_stats
    description: "Daily statistics by language"
    columns:
      - name: stat_date
        description: "Date of the statistics"
        tests:
          - not_null

Update dbt_project.yml

name: 'github_analytics'
version: '1.0.0'

profile: 'default'

model-paths: ["models"]
test-paths: ["tests"]

target-path: "target"
clean-targets:
  - "target"
  - "dbt_packages"

models:
  github_analytics:
    staging:
      +schema: staging
      +materialized: view
    marts:
      +schema: marts
      +materialized: table

Running dbt

In dbt Cloud:

1. Open the IDE

2. Click “Run” or type dbt run

3. Watch the models build

4. Click “Test” or type dbt test

5. Check for failures

You can also create a job to run daily:

1. Go to Jobs

2. Create a new job

3. Add command: dbt run && dbt test

4. Schedule: After Fivetran completes (e.g., 2 AM daily)

5. Enable notifications

Part 8: Dashboards with Looker Studio

Why Looker Studio?

Looker Studio (formerly Google Data Studio) is free. Forever. No trial period.

It connects directly to Snowflake. It’s easy to learn. And it looks professional.

Setting Up Looker Studio

Step 1: Open Looker Studio

1. Go to lookerstudio.google.com

2. Sign in with your Google account

Step 2: Connect to Snowflake

1. Click “Create” → “Data Source”

2. Search for “Snowflake”

3. Click “Authorize” to install the connector

4. Enter your Snowflake details:

   • Account identifier: Your Snowflake account

   • Username: DBT_USER

   • Password: Your password

   • Warehouse: TRANSFORM_WH

   • Database: GITHUB_ANALYTICS

   • Schema: MARTS

5. Choose a table (start with fct_daily_stats)

6. Click “Connect”

Step 3: Create a dashboard

1. Click “Create Report”

2. Add your data source

3. Start adding charts

Dashboard Ideas

Dashboard 1: Language Popularity

Charts to include:

• Pie chart: Total repos by language

• Bar chart: Total stars by language

• Line chart: Stars over time

• Scorecard: Total repositories

Dashboard 2: Top Repositories

Charts to include:

• Table: Top 20 repos by stars

• Bar chart: Repos with most forks

• Scatter plot: Stars vs forks relationship

Dashboard 3: Activity Trends

Charts to include:

• Line chart: New repos per week

• Bar chart: Most active languages

• Table: Recent updates

Tips for Good Dashboards

1. Keep it simple. Don’t add too many charts.

2. Use consistent colors.

3. Add titles that explain what the chart shows.

4. Put the most important insights at the top.

5. Add date filters so users can explore.

Part 9: What You Built

Let’s recap. You now have:

This is a real data engineering solution. Companies use the same architecture. The same tools. The same patterns.

And we didn’t touch a single line of Python! Imagine a Senior Data Engineer will earn 200k US or 180k CAD for the same setup.

The key is not the tools, but the system design basics.

Part 10: What’s Next?

You have a working project. But there’s always more to learn. Here are the next steps to make your project better.

Add Data Quality Tests

dbt has built-in tests. Add more:

# In your yml files

columns:

  - name: stars

    tests:

      - not_null

      - dbt_utils.accepted_range:

          min_value: 0

Add Documentation

dbt generates documentation automatically:

dbt docs generate

dbt docs serve

This creates a website showing all your models and their relationships.

Implement SCD Type 2

Use dbt snapshots to track history:

{% snapshot repositories_snapshot %}

{{

    config(

      target_schema=’snapshots’,

      unique_key=’repository_id’,

      strategy=’timestamp’,

      updated_at=’updated_at’,

    )

}}

SELECT * FROM {{ source(’github_raw’, ‘repository’) }}

{% endsnapshot %}

Add CI/CD

Set up GitHub Actions to test your dbt code on every pull request.

Add Infrustructure as a Code

Try to use Terraform for your Snowflake or cloud data infrastructure.

Add the Orchestration tool

Add Airflow or a similar tool, such as Dagster, Prefect or you can go all in on Python and run AWS Lambdas to ingest data.

Monitor Costs

Snowflake charges for compute time. Monitor your credits:

1. Go to Admin → Cost Management

2. Set up alerts before credits run out

3. Use smaller warehouses when possible

Add More Data Sources

Once you’re comfortable, add more sources:

• A PostgreSQL database

• A Google Sheets file

• A Jira or Notion integration

Each new source teaches you something new.

Key Takeaways

1. You can learn data engineering for free. Enterprise tools offer generous trials.

2. You don’t need a cloud account. Snowflake, dbt Cloud, and Fivetran work on their own.

3. This is production-ready architecture. Companies use the same patterns.

4. Dimensional modelling matters. Learn facts, dimensions, and star schemas.

5. Know your SCDs. Type 1 vs Type 2 is an interview favourite.

6. Start simple. Get it working first. Optimize later.

7. This project shows real skills. Put it on your resume. Talk about it in interviews.

Final Words

You now know how to build a real data engineering project. For free. Using enterprise tools.

This is how modern data teams work. Source data flows through ingestion, lands in storage, gets transformed, and powers dashboards. The same pattern. Every time.

Build this project. Put it on GitHub. Add it to your resume. Talk about it in interviews.

You’ve got this.

Python is the number one language for data jobs. Whether you’re aiming for data engineering, data analytics, machine learning, or generative AI - Python is everywhere.

But here’s the thing: Python is big. Really big. And complicated.

Where do you even start?

If you’re new to Python, the amount of information can feel overwhelming. If you’re experienced, you might wonder: “Do I really need to know everything?”

The answer is no. You don’t need to master everything. You need just enough Python for data roles.

This guide will show you what to learn, how to learn it (hint: books, not just videos), and what you can skip. I’ll share my personal learning method that helped me build a 15+ year career in data engineering.

Let’s start with something important.

“The single most important skill for a computer scientist is problem solving. Problem solving means the ability to formulate problems, think creatively about solutions, and express a solution clearly and accurately.” - Allen B. Downey

This applies to data roles too. You might work as a data engineer, data analyst, or data scientist. But always focus on the problem first. The problem should come from the business. We shouldn’t just write code because we like it. We need to solve real business problems and add value.

Keep that in mind as you learn Python. The language is a tool. The problem-solving mindset is what matters.

🐍 Why Python? (The Reality Check)

Python’s Dominance

Python is everywhere in data. It’s the most popular language for data jobs. You can use it for:

• Data analytics and exploration

• Data engineering pipelines

• Machine learning models

• Generative AI applications

• Cloud infrastructure and automation

The community is huge. There are thousands of packages and libraries ready to use. And with AI assistants like Cursor and Copilot, you can contribute to Python codebases even with basic knowledge.

When Python Isn’t Enough

Python has limits. It’s not always the best choice for:

• Production applications at massive scale (millions of users)

• Performance-critical systems

• Some real-time applications

But for data roles? Python is the standard. It’s what you need.

Python Alternatives (Quick Reality Check)

You might hear about other languages:

• Scala: Popular for Spark and Hadoop. But you can use PySpark instead.

• R: Some legacy code exists. But Python is the modern choice.

• Rust/C++: High performance, but smaller data community.

Bottom line: Focus on Python first. Learn alternatives only if your company uses them. With AI assistants, you can adapt to other languages when needed.

📚 The Learning Method: Why Books Beat Tutorials

Here’s my personal take: Books are better than video tutorials for learning Python.

The Problem with Video Tutorials

Video tutorials are:

• Too fast-paced. Not enough time to digest concepts.

• Hard to return to. Finding that one explanation you need is frustrating.

• Limited practice. You watch, but don’t always do.

• Often skip fundamentals. They rush to the “cool stuff.”

Why Books Work Better

Books let you:

• Learn at your own pace. Read, pause, re-read. No one is rushing you.

• Practice with exercises. Hands-on learning beats passive watching.

• Create notes and summaries. Active learning helps you remember.

• Return to concepts. Can’t remember something? Flip back a few pages.

Here’s a technique that helped me: Write down Python scripts and learn them by heart. Try to understand what each line means. Then review them during your commute or breaks. This repetition builds understanding.

Recommended Books

I’ve read many Python books. Here are two I highly recommend:

1. Think Python (Free!)

Why it’s great: Simple, clear, perfect starting point.

Where to get it: Available for free online. Just search “Think Python free download.”

Best for: Complete beginners who want a solid foundation.

Link: https://greenteapress.com/thinkpython2/thinkpython2.pdf

2. Python Crash Course

Why it’s great: Two-part structure that’s brilliant.

• Part 1: Fundamentals. Functions, classes, variables, data types. Everything you need to know.

• Part 2: Real-world projects. You build a website with Django, create a game, build a data analytics solution. You apply what you learned.

Best for: Learning fundamentals AND seeing how to build complete applications.

Unique value: Most books teach concepts. This one shows you how to use them end-to-end.

The “Three Books Rule”

A friend gave me great advice: “If you want to learn something, read at least three books on that topic.”

Don’t waste time on endless video tutorials. Go to Amazon, check the best-reviewed Python books, and get them. Start with Think Python or Python Crash Course. Then find a third book that covers areas you want to dive deeper into.

🎯 Core Python Fundamentals: The 5 Essentials

You don’t need to learn everything about Python. You need to master these five things first.

1. Variables & Data Types

What: The basic building blocks.

Types you need to know:

• int - whole numbers (1, 2, 100)

• float - decimal numbers (3.14, 2.5)

• string - text (”hello”, “data”)

• boolean - True or False

• None - represents nothing

Why it matters: Everything in Python is built on these. You can’t skip this.

Practice: Create variables. Convert between types. Understand what each type means.

2. Data Structures

What: How you organize and store data.

The four you need:

• Lists: Ordered collections you can change. [1, 2, 3]

• Tuples: Ordered collections you can’t change. (1, 2, 3)

• Dictionaries: Key-value pairs. {”name”: “John”, “age”: 30} - Most important for data work

• Sets: Collections of unique elements. {1, 2, 3}

Why it matters: This is how you work with data. Lists and dictionaries are everywhere in data roles.

Practice: Create each type. Add items, remove items, access values.

3. Control Flow

What: How you make decisions and repeat actions.

The basics:

• if/else statements: Decision making. “If this is true, do that.”

• for loops: Repeat something for each item in a list.

• while loops: Repeat something while a condition is true.

Why it matters: This is the logic of your code. Without this, you can’t automate anything.

Practice: Write if statements. Loop through lists. Combine conditions.

4. Functions

What: Reusable blocks of code.

Key concepts:

• def keyword - how you create functions

• Parameters and arguments - passing data into functions

• Return values - getting results back

• Lambda functions - quick one-liners

Why it matters: Functions let you write code once and use it many times. Essential for any real project.

Practice: Write simple functions. Pass data in. Get results out. Use functions in other functions.

5. Modules & Packages + Virtual Environments

What: Using other people’s code and managing dependencies.

Key concepts:

• import statements - using existing code

• Creating modules - organizing your own code

• Virtual environments - isolating dependencies (avoiding “dependency hell”)

Why it matters: You’ll use libraries like pandas, requests, and boto3. You need to know how to import them. Virtual environments are critical for professional development.

Practice: Import standard library modules. Install packages with pip. Create a virtual environment.

Nice to Have (But Not Critical Yet)

• List Comprehensions: Elegant one-liners. Useful for interviews, but not essential for daily work.

• Exception Handling: try/except/finally blocks. Good practice, but you can learn this later.

Learning strategy: Master the 5 essentials first. If you don’t understand something, take a break and revisit it. Don’t rush. Understanding beats speed.

📊 Key Topics for Data Roles

Now that you have the fundamentals, here are the seven areas that matter for data roles:

1. Data Manipulation - Working with data

2. Data Processing - Processing large datasets

3. Database Connectivity - Connecting to databases

4. Data Validation & Quality - Ensuring data is correct

5. Cloud & Infrastructure - Working with cloud services

6. Data Formats & Serialization - Reading and writing different file types

7. Advanced Concepts - Optional but valuable

Let’s dive into each one.

📈 1. Data Manipulation

Pandas (The Workhorse)

What it is: Pandas lets you work with DataFrames and Series. Think of it as Excel or SQL tables, but in Python.

Key operations you’ll use:

• read_csv() - Reading CSV files

• groupby() - Grouping and aggregating data

• merge() - Joining data (like SQL JOINs)

• pivot_table() - Reshaping data

Use cases: Data analysis, exploration, ad-hoc requests.

Best for: Data Analysts and Data Scientists.

Important limitation: Pandas is great for notebooks and local analysis. It’s not ideal for production pipelines. Data engineers often use SQL or Spark instead.

Reality check: If you’re comfortable with SQL, you might use DuckDB and solve many problems without deep Pandas knowledge. But if your organization uses Python, you need to understand their code.

NumPy

What it is: Arrays and mathematical operations.

Why it matters: Pandas is built on NumPy. Many ML libraries use it too.

Use cases: Mathematical computations, working with arrays.

Polars

What it is: A faster alternative to Pandas.

When to use it: If Pandas is too slow for your dataset.

Reality: Most people start with Pandas. Learn Polars if you hit performance issues.

PyArrow

What it is: Support for columnar data formats.

Why it matters: Efficient data storage and transfer, especially for Parquet files.

⚙️ 2. Data Processing

PySpark (The Big Data Tool)

What it is: Distributed data processing. Process data across multiple machines.

Key concepts:

• RDDs (Resilient Distributed Datasets)

• DataFrames

• Transformations and actions

Platforms: Apache Spark, Databricks

Reality check: Many companies use Spark SQL instead of PySpark. You might not need deep PySpark knowledge if SQL works for your use case.

When you need it: If your company uses Spark and you need to contribute Python code.

Dask

What it is: Parallel computing for larger-than-memory datasets.

Reality: Less commonly used. Good to know it exists.

When: If your organization uses it.

Apache Airflow (The Orchestrator)

What it is: A framework for scheduling and running data pipelines.

Key concepts:

• DAGs (Directed Acyclic Graphs) - your pipeline structure

• Operators - tasks in your pipeline

• Tasks - individual steps

Why it’s critical: Extremely popular in data engineering. You’ll likely encounter it.

Python knowledge needed:

• You don’t need to be a Python expert

• You need to understand code (read functions, understand variables)

• Often template-based (copy, paste, modify parameters)

Covered in: Surfalytics Module 4 of our course (if you’re following along).

🗄️ 3. Database Connectivity

When You Need It

You’ll need database connectivity when:

• Extracting data from databases

• Writing data to databases

• Building end-to-end data solutions

Key Libraries

• SQLAlchemy: ORM (Object-Relational Mapping). Works with many databases.

• psycopg2: PostgreSQL adapter. Direct connection to PostgreSQL.

• pymongo: MongoDB driver. For NoSQL databases.

• pyodbc: ODBC connections. Works with various databases.

The Reality

Alternative tools exist: Fivetran, Airbyte (no-code/low-code tools). You might not need Python for simple extractions.

When to use Python:

• Custom extraction logic

• No connector available

• Contributing to the existing Python codebase

Practice: We’ll have examples with PostgreSQL in Module 2.

✅ 4. Data Validation & Quality

Why It Matters

Data quality is critical. Bad data leads to bad decisions. Production systems need validation.

Key Tools

• Pydantic: Data validation using Python type hints. Great for API data.

• Great Expectations: Data quality testing framework. Comprehensive validation.

• Pandera: DataFrame validation. Check your pandas DataFrames.

• pytest/unittest: Unit testing. Test your functions.

• DBX: Databricks framework for data quality.

Advanced Concept: Test-Driven Development (TDD)

What it is: Write tests before writing functions.

Why it helps: Ensures your code works correctly from the start.

When: More advanced use case. Shows professional maturity.

☁️ 5. Cloud & Infrastructure

AWS (boto3)

What it is: AWS SDK for Python. The official way to interact with AWS services.

Use cases:

• S3 (storage)

• Lambda (serverless functions)

• Glue (ETL jobs)

• Triggering jobs

• Creating resources

Why it matters: AWS is the most popular cloud platform.

Google Cloud

google-cloud-storage: GCP integration. Similar patterns to AWS.

Azure

azure-storage-blob: Azure integration. Similar patterns to other clouds.

Docker

What it is: Containerization. Package your application with all dependencies.

Why it matters: Ensures your code runs the same way everywhere.

Connection: Often used with Python applications in production.

API Requests (Critical!)

requests library: Connecting to REST APIs.

Why it’s critical: Many data sources come from APIs. You’ll use this a lot.

Typical workflow:

1. Connect to API endpoint

2. Get JSON data

3. Process and manipulate data

4. Save to database or files

Connection: Often requires database connectivity libraries too. It all connects.

Learning Strategy

Build 5-6 pet projects. This covers 20% of knowledge but 80% of use cases.

Project ideas:

• API → Process → Database

• File processing pipeline

• Data validation script

• Cloud storage interaction

• Simple Airflow DAG

Build end-to-end: Don’t just learn concepts. Build complete solutions.

📦 6. Data Formats & Serialization

Why It Matters

Different systems use different formats. You need to read and write various file types.

Key Formats

• JSON: json.loads(), json.dumps(). API responses, configuration files.

• CSV: csv module, pandas.read_csv(). Spreadsheet data.

• Parquet: Columnar format (pyarrow, fastparquet). Efficient storage. Very popular in data engineering.

• Avro: Data serialization. Used in big data systems.

Learning Approach

• Understand when to use each format

• Practice reading and writing each type

• Know the trade-offs (size, speed, compatibility)

🚀 7. Advanced Concepts (Optional but Valuable)

When to Learn

• After mastering fundamentals

• When you need them for specific use cases

• To show depth in interviews

Key Concepts

Decorators: @decorator syntax, function wrappers.

Reality: You rarely create your own. Tools provide them (Datadog, Duer, etc.). But understanding them helps.

Generators: yield keyword, memory-efficient iteration.

Context Managers: with statement, file handling. You’ve probably used this: with open(’file.txt’) as f:

Type Hints: Static typing for better code quality. Helps with IDE support and catching errors.

Async/Await: Asynchronous programming for I/O operations. Advanced topic.

Logging: logging module - CRITICAL for production.

Why logging matters:

• Visibility into running programs

• Debugging when things go wrong

• Monitoring and alerting

• Failure analysis

When you need it: Airflow DAGs, data transformation jobs, any production code.

Value: Without logging, your code is a black box. With logging, you can see what’s happening.

🎓 Best Practices (The Professional Touch)

These show you’re a professional, not just someone who writes code.

PEP 8 (Style Guide)

What it is: Python code style standards.

Why it matters: Readability, consistency, professionalism.

Tools: Linters can enforce this automatically.

Version Control (Git)

Why it’s essential: Collaboration, history, rollback.

Not Python-specific: But critical for data engineers.

Best practice: Use Git for all code. Every project.

CI/CD

What it is: Continuous Integration/Deployment. Automated testing and deployment.

Why it matters: Catches errors early. Deploys consistently.

Advantage: Software engineering practices in data = competitive edge.

When: Start early. The sooner the better.

Environment Variables

What: os.environ, python-dotenv

Why it’s CRITICAL: Secure credential management.

Security rule: Never hardcode passwords in code. Never share credentials with LLMs.

Best practice: Use environment variables for all secrets.

Configuration Management

Formats: YAML, TOML, ConfigParser

Why: Separate configuration from code.

Benefit: Easy to change settings without code changes.

📦 Package Managers & Environment Managers

The Problem: Dependency Hell

Different packages can conflict. You need isolated environments per project. Teams need reproducible environments.

Package Managers

What they do: Install, update, manage Python packages.

Top 3:

1. pip - Default Python package installer (most common)

2. uv - Ultra-fast, Rust-based (modern alternative)

3. Poetry - Advanced dependency resolution (popular in data teams)

Environment Managers

What they do: Create isolated Python environments.

Top 3:

1. venv - Built-in Python tool

2. Conda - Cross-platform (environment + package manager)

3. pyenv - Manages multiple Python versions

Combined Solutions

• Poetry: Package manager + environment manager

• uv: Package manager + environment manager

Strategy: Pick one. Learn it well. Use it consistently.

Best Practice

• Every project (GitHub repo) should have its own environment

• Define package versions for reproducibility

• Goal: Code runs on any machine (colleague’s, cloud, Airflow)

🎯 Learning Roadmap: The Practical Path

Here’s a step-by-step path from beginner to job-ready.

Phase 1: Foundations (Weeks 1-2)

1. Read: Think Python or Python Crash Course (Part 1)

2. Master: The 5 core fundamentals

3. Practice: Write simple scripts. Do exercises from the book.

4. Goal: Understand variables, data structures, control flow, functions

Don’t rush this phase. If you don’t understand something, take a break and revisit it.

Phase 2: Data Basics (Weeks 3-4)

1. Learn: Pandas basics (read, filter, group, merge)

2. Practice: Work with CSV files. Do data analysis exercises.

3. Learn: Virtual environments and package management

4. Goal: Can manipulate data in Python

Focus: Get comfortable with DataFrames. This is where you’ll spend a lot of time.

Phase 3: Real Projects (Weeks 5-8)

1. Build: 5-6 pet projects covering key use cases

   • API → Process → Database

   • File processing pipeline

   • Data validation script

   • Cloud storage interaction

   • Simple Airflow DAG

2. Learn: As you build (database connectivity, API requests, etc.)

3. Goal: End-to-end data solutions

Key: Learn by doing. Don’t just read. Build.

Phase 4: Production Skills (Ongoing)

1. Learn: Logging, error handling, testing

2. Practice: Git, CI/CD basics

3. Read: Best practices, style guides

4. Goal: Professional-grade code

This never ends. Keep learning and improving.

🎯 Key Takeaways

Here’s what to remember:

1. Problem-solving first: Focus on business problems, not just code

2. Books over videos: Learn at your own pace with books

3. Master the 5 essentials: Variables, data structures, control flow, functions, modules/environments

4. Build pet projects: 5-6 pet projects cover 80% of use cases

5. Just enough Python: You don’t need to master everything, just what’s needed for data roles

6. Tools matter more: After Python basics, focus on data tools (Snowflake, Airflow, dbt)

7. Professional practices: Git, CI/CD, logging, environment variables - start early

🚀 What’s Next?

Ready to start? Here’s your action plan:

1. Get a book: Start with Think Python (free) or Python Crash Course

2. Build your first pet project: API → Process → Database (Data Engineering System Design?)

3. Share your journey: Learning in public helps you and others

4. Join the community: Get support, ask questions, help others at Surfalytics.com

Remember: You don’t need to master everything. Just enough Python to solve data problems. Focus on the fundamentals, build projects, and keep learning.

The journey is long, but it’s worth it. Python opens doors in data roles. Start today.

📎 Common Questions

Do I need Python for my first data job?

Not always mandatory, but highly recommended. Many jobs list Python as “nice to have” or “preferred.” But the more you know, the more opportunities you have. All jobas are required decent knowledge of SQL.

How long does it take to learn?

2-3 months of focused learning for basics. But learning never stops. You’ll keep improving throughout your career.

It will require daily/weekly practice. The most important thing is to use Python in the context of data analytics and data engineering projects.

What if my company uses Scala/R?

Learn Python first. Then adapt. With AI assistants, you can contribute to other languages. But Python is the standard for data roles.

Can AI help me learn?

Yes, but understand fundamentals first. AI can help you write code, but you need to understand what it’s doing. Start with books and practice.

The 80/20 Rule

20% of Python knowledge = 80% of data role use cases.

Focus on: Data manipulation, database connectivity, API requests, basic processing.

Learn advanced concepts only when needed.

🎢 Real use cases

Let’s look at three real-world data analytics solutions. Each one uses Python differently. This will help you understand when Python is essential and when you can get by without it.

Use Case 1: The Gentle Data Analytics Solution (No Python)

The Setup:

• Data Ingestion: Fivetran (no-code tool) extracts data from various APIs

• Data Warehouse: Snowflake stores all the data

• Data Transformation: dbt transforms data using SQL

• BI Tool: Looker connects to Snowflake for dashboards and reports

Python Usage: 0% - No Python at all, except SQL.

What This Means: This is a completely no-code solution (except SQL). You don’t need Python for this setup. Fivetran handles API connections. Snowflake stores everything. dbt does transformations in SQL. Looker visualizes the data.

When This Works:

• Standard data sources with existing connectors

• Team comfortable with SQL

• No custom extraction logic needed

• Fast time-to-value

The Reality: Many companies start here. It’s simple, maintainable, and gets the job done. You can build a complete data analytics solution without writing a single line of Python.

Takeaway: Python isn’t always required. SQL and no-code tools can solve many problems.

Use Case 2: The Airflow-Powered Solution (Python for Orchestration)

The Setup:

• Data Ingestion: Airflow runs Python jobs to extract data from APIs

• Data Warehouse: Snowflake stores the data

• Data Transformation: Airflow orchestrates dbt runs (SQL transformations)

• Orchestration: Airflow schedules and coordinates everything

Python Usage: Moderate - Python for data extraction, API calls, and orchestration.

What This Means: Python is important here. You write Python scripts in Airflow to:

• Extract data from APIs (using requests library)

• Handle authentication and API pagination

• Transform data before loading (if needed)

• Orchestrate the entire pipeline

• Handle errors and retries

When This Works:

• Custom API connections not available in Fivetran

• Need more control over extraction logic

• Complex data transformations before loading

• Want to orchestrate everything in one place

The Reality: This is common in data engineering. Python stitches everything together. You need to understand Python code, even if you’re not writing complex algorithms. You’re reading functions, modifying parameters, and understanding data flow.

Takeaway: Python is important for data extraction and orchestration. You don’t need to be an expert, but you need to understand code.

Use Case 3: The Databricks PySpark Solution (Python-Heavy)

The Setup:

• Data Ingestion: PySpark jobs extract data from S3 (AWS storage)

• Data Processing: Databricks processes data using PySpark

• Architecture: Medallion architecture (Bronze → Silver → Gold layers)

• BI Tool: Tableau connects to Gold layer (no Python)

• Streaming Analytics: Custom web app runs near real-time analytics on Bronze data

Python Usage: Heavy - Almost everything is Python/PySpark. Almost no SQL.

What This Means: This is a Python-heavy environment. You’re using PySpark for:

• Reading data from S3

• Transforming data across Bronze, Silver, and Gold layers

• Complex data processing and aggregations

• Building streaming analytics solutions

• Custom business logic

The Architecture:

• Bronze Layer: Raw data from S3 (PySpark reads and stores)

• Silver Layer: Cleaned and validated data (PySpark transformations)

• Gold Layer: Business-level aggregates (PySpark aggregations)

• Streaming: Real-time processing on Bronze data (PySpark Streaming)

When This Works:

• Large-scale data processing (petabytes)

• Complex transformations that SQL can’t handle easily

• Need distributed processing

• Real-time or near real-time requirements

• ML workloads alongside analytics

The Reality: This is advanced data engineering. You need strong Python skills. You’re writing PySpark code, understanding distributed computing, and building complex pipelines. SQL is minimal here.

Takeaway: Some roles require deep Python knowledge. PySpark is Python, and you need to be comfortable with it.

What These Use Cases Teach Us

Python Usage Spectrum:

1. No Python: SQL + no-code tools work for many solutions

2. Some Python: Needed for orchestration and custom extraction

3. Heavy Python: Required for distributed processing and complex transformations

The Key Insight: Different companies need different levels of Python. Some roles require almost no Python. Others require deep expertise. Most fall somewhere in between.

Your Learning Path:

• Start with fundamentals (the 5 essentials)

• Build projects that match your target role

• If you want data engineering roles, learn Airflow and PySpark

• If you want analytics roles, SQL might be enough (but Python helps)

The Bottom Line: Understand where Python fits in real solutions. Sometimes it’s essential. Sometimes it’s optional. But having Python skills opens more doors.

The 45-Minute Challenge

Imagine you’re sitting in a System Design interview. The interviewer just gave you a couple of sentences about requirements, and now you have around 45 minutes to dive deep into the architecture of a data engineering system. Your whiteboard is blank. The clock is ticking. ⏰

Sound familiar?

Whether you’re prepping for an interview or just want to level up your architectural thinking, this guide will give you a framework to tackle any data engineering system design challenge with confidence.

Let’s break it down.

🗺️ The Master Architecture

Before we dive into the details, here’s the high-level view of what we’re building. A solid data engineering system has six essential layers:

Think of this as your mental model. Every system design conversation should touch on these layers, though you’ll spend more time on some than others depending on the requirements.

📊 Sources Layer
    ↓
⚙️ Processing Layer
    ↓
💾 Storage Layer
    ↓
🤖 ML/AI Layer (Optional)
    ↓
👥 Consumers Layer
    ↓
🛠️ Tools Layer (Cross-cutting)

🎬 Where Do You Start?

First of all, you can decide where to start. From top to bottom or bottom to top?

Here’s the truth: It doesn’t really matter for data engineering architecture. You can start with sources and work your way down, or start with consumers and work backwards. Both approaches work.

However, there’s one exception: data modelling interviews. If you’re specifically being asked about data modelling, start from the consumer requirements and work backwards. This shows you’re thinking about the business use case first.

💡 Pro Tip: Tell a Story, Don’t Just Draw Boxes

Use a drawing board like Drawio, Lucidchart, or at least Google Slides. Tell the story of how data flows through your system. Don’t just list components—explain why each piece exists and how they connect.

The best candidates I’ve interviewed don’t just draw architecture; they narrate the data journey. “First, we capture events from the application, then we...” That storytelling makes all the difference.

📚 The Six Essential Layers (Deep Dive)

1️⃣ Source Layer: Where It All Begins

This is where you show you understand how businesses actually generate data. Start by listing the available sources based on what the interviewer told you.

Common source types:

• 🗄️ OLTP Databases - PostgreSQL, MySQL, SQL Server (transactional systems)

• 📁 Files - CSV, JSON, Parquet on SFTP, S3, Azure Blob

• 🌐 APIs - REST APIs, webhooks, third-party SaaS tools

• 📨 Event Streams - Kafka, RabbitMQ, AWS Kinesis, Azure Event Hubs

• 📱 IoT Devices - Sensor data, mobile apps, edge devices

Questions you MUST ask:

1. What’s the data volume? Are we talking about gigabytes or petabytes? This drives your storage and processing decisions.

2. What are the latency requirements? Do users need data in seconds, minutes, or hours? This determines batch vs stream processing.

3. What data formats are we dealing with? Structured (tables), semi-structured (JSON), or unstructured (images, documents)?

4. How is the business generating this data? Understanding the business context is crucial. Is this customer data? Transaction logs? Clickstream events?

5. Do we have CDC (Change Data Capture)? If the source is a database, can we capture incremental changes, or do we need full table dumps?

*Why this matters: The source layer defines your constraints. If you have real-time event streams, you can’t just batch process them once a day. If you’re pulling from a legacy SFTP server with nightly file drops, streaming might be overkill.

Talk through the sources. Ask questions. Show curiosity. This demonstrates you’re not just memorizing patterns—you’re thinking about real systems.

2️⃣ Storage Layer: Where You Show Your Knowledge 🎓

This is your moment to shine. Storage decisions are your primary responsibility as a data engineer. This is where you separate the junior engineers from the senior ones.

The big question: Where do we store the data?

The Three Main Options:

🏢 Data Warehouse

• Examples: Snowflake, BigQuery, Redshift, Synapse

• Best for: Structured data, SQL analytics, BI reporting

• Pros: Optimized for queries, easy to use, great for business users

• Cons: Can be expensive, less flexible for ML workloads, vendor lock-in

🏞️ Data Lake

• Examples: Storage in S3, Azure Data Lake Storage, Google Cloud Storage with separated compute like Spark, Trino, Athena, EMR, Dataproc, Synapse and etc.

• Best for: Large-scale raw data, unstructured data, cost-effective storage

• Pros: Cheap storage, flexible, supports any format

• Cons: Can become a “data swamp,” requires more engineering effort

🏰 Lakehouse

• Examples: Databricks (Delta Lake), Snowflake (Iceberg), Dremio, Trino

• Best for: Combined analytics + ML workloads, flexible architecture

• Pros: Best of both worlds, ACID transactions on data lake, unified platform

• Cons: Newer paradigm, some tools still maturing

How to Choose?

Ask these questions:

1. What public cloud are you using? Azure, AWS, or GCP? Each has native options that integrate well.

2. Do you already have a data warehouse? Don’t reinvent the wheel if something is working.

3. What are the query patterns?

   1. Mostly SQL reporting → Warehouse

   2. Mostly ML/data science → Lake/Lakehouse

   3. Both → Lakehouse

4. What’s the data volume and growth rate? Petabyte-scale often favors lake/lakehouse for cost reasons.

⚠️ Important:There is no wrong answer. Each approach has trade-offs. What matters is that you can justify your choice based on the requirements.

Architecture Patterns to Mention

Medallion Architecture (Bronze → Silver → Gold) - This is especially popular in lakehouse architectures:

• Bronze (Raw): Land data exactly as received. Immutable, append-only.

• Silver (Cleaned): Deduplicated, validated, standardized formats.

• Gold (Business-level): Aggregated metrics, dimensional models, ready for consumption.

Why it’s great: Clear separation of concerns, easy to troubleshoot, supports both batch and streaming.

Dimensional Modeling (Star/Snowflake Schema) - Still relevant for traditional data warehouse approaches:

• Fact tables (events, transactions)

• Dimension tables (customers, products, dates)

• Optimized for BI tools and reporting

Don’t dismiss this as “old school.” Many business users still need simple, well-defined dimensional models.

Data Transformation Strategy

The question: How will you transform the data?

The default answer: dbt (SQL-based transformation) 🎯

Why dbt is often the right choice:

• ✅ SQL is accessible to more team members

• ✅ Built-in testing and documentation

• ✅ Version control and CI/CD friendly

• ✅ Strong community and ecosystem

• ✅ Works with all major warehouses and lakehouses

Alternatives to mention:

• Spark (PySpark/Scala) - For large-scale transformations, complex logic

• Python/Pandas - For ML feature engineering, custom logic

• Stored Procedures - Legacy systems (mention you’d migrate away from these)

Pro tip: Start with SQL/dbt. Only reach for Spark when you truly need it. Complexity should be justified.

3️⃣ Consumers Layer: Start With the End in Mind 👥

This is how the data insights will be used. Understanding your consumers drives everything else.

Who are the target audiences?

1. 📊 Business Analysts

   1. Tools: Tableau, Power BI, Looker, Mode

   2. Needs: Pre-aggregated tables, dimensional models, fast queries

   3. Frequency: Daily dashboards, ad-hoc queries

2. 🔬 Data Scientists

   1. Tools: Jupyter notebooks, Python, R

   2. Needs: Raw and feature-engineered datasets, ML-ready formats

   3. Frequency: Varies by project phase

3. 🤖 Applications

   1. Tools: REST APIs, GraphQL, Reverse ETL tools

   2. Needs: Low-latency access, real-time or near-real-time data

   3. Frequency: Continuous, high QPS (queries per second)

4. 🤝 External Partners

   1. Tools: Data shares (Snowflake), secure APIs, file exports

   2. Needs: Governed, secure access to specific datasets

   3. Frequency: Scheduled or on-demand

5. 🧠 GenAI Applications

   1. Tools: Vector databases, RAG pipelines, LLM APIs

   2. Needs: Embeddings, semantic search, context retrieval

   3. Frequency: Real-time or batch depending on use case

Critical questions to ask:

• How frequently do they need data? Every 15 minutes? Once an hour? Once a day? This determines your SLA and processing cadence.

• What’s the acceptable data latency? There’s a big difference between “yesterday’s data” and “last 5 minutes of data.”

• What format do they prefer? SQL-accessible tables? JSON via API? CSV files? Parquet on S3?

• What’s the scale? How many users? How many queries per minute?

Why this matters: Consumer requirements drive your architecture decisions. If your BI team needs data once a day, you can batch process overnight. If your fraud detection app needs real-time data, you’re building a streaming pipeline.

The best engineers I’ve worked with always start by understanding the consumer needs. That’s how you avoid over-engineering (or under-engineering).

4️⃣ Processing Layer: Batch vs Stream ⚙️

This is how we move data from the Source Layer to the Storage Layer.

There are fundamentally only two options: Batch and Stream.

🔄 Batch Processing (The Default Choice)

Why batch is great:

• ✅ Simple to understand and maintain

• ✅ Easy to debug and replay

• ✅ Cost-effective (run when you need to)

• ✅ Well-established patterns and tools

• ✅ Handles large historical data well

Common tools:

• Airflow (orchestration)

• dbt (transformation)

• Spark (heavy lifting)

• Fivetran/Airbyte (managed ingestion)

Typical patterns:

• Full refresh: Replace entire table (simple but inefficient)

• Incremental loading: Only process new/changed records

• Time-based partitioning: Process data by date ranges

When to use batch:

• Daily reporting and analytics

• Historical data analysis

• When near-real-time isn’t required

• When simplicity is more important than latency

🌊 Stream Processing (When You Need It)

When streaming is required:

• Real-time dashboards and alerting

• Fraud detection (sub-second response)

• IoT sensor data (constant flow)

• Application events (clickstream, user activity)

• Financial transactions (immediate reconciliation)

Common tools:

• Kafka / Confluent (event streaming)

• Spark Streaming / Structured Streaming

• Flink (low-latency complex event processing)

• AWS Kinesis / Azure Event Hubs (managed services)

The hybrid approach:

This is actually very common in real systems:

1. Stream events into cloud storage (S3, ADLS)

2. Process from storage in micro-batches

3. Get near-real-time results without full streaming complexity

Why this works: You get durability and replay capability from storage, without the operational complexity of true streaming.

🎯 Critical Concepts to Mention (Show You Know Production)

1. Idempotency

Your pipeline should produce the same result if you run it multiple times. Running the same job twice shouldn’t create duplicate records.

Example: Use MERGE/UPSERT instead of INSERT. Key on unique business keys.

2. CDC (Change Data Capture)

Capture only what changed in the source system instead of pulling full tables every time.

Tools: Debezium, AWS DMS, Fivetran, Airbyte

3. Incremental Loading

Only process new or changed data. Track watermarks, timestamps, or sequence IDs.

-- Example: Load only records modified since last run

WHERE updated_at > (SELECT MAX(updated_at) FROM target_table)

4. Data Quality Checks

• Validate data at ingestion and transformation stages:

• Schema validation (expected columns present?)

• Null checks (required fields populated?)

• Range checks (values within expected bounds?)

• Referential integrity (foreign keys valid?)

• Freshness checks (data arriving on time?)

Tools: Great Expectations, dbt tests, custom validation

5. Schema Evolution

How do you handle when source systems add/remove/rename columns?

Strategies:

• Schema-on-read (flexible but risky)

• Schema validation (strict but safe)

• Graceful degradation (log issues, don’t fail)

💡 Pro tip: Batch is a great default choice. Lead with that. Then say, “But for use cases like [specific example from requirements], we’d need streaming.”

Don’t over-engineer. Streaming adds complexity. Only use it when latency truly matters to the business.

5️⃣ ML/AI Layer: The Optional But Impressive Section 🤖

Here’s the secret: No one expects you to cover this, but if you do, you’ll stand out.

This shows you understand modern data platforms aren’t just for reporting—they’re powering machine learning and AI applications.

Components to Mention

🗃️ Feature Store

A centralized repository for ML features that serves both training (batch) and inference (real-time).

• Offline features: Historical data for model training

• Online features: Low-latency access for real-time predictions

• Tools: Feast, Tecton, SageMaker Feature Store, Databricks Feature Store

Why it matters: Prevents feature engineering duplication. Data scientists define features once, use them everywhere.

🧪 ML Models

Where do models live in your architecture?

• Training pipelines: Scheduled jobs to retrain models

• Model registry: Version control for models

• Model serving: Batch predictions vs real-time API endpoints

Tools: MLflow, Kubeflow, SageMaker, Azure ML

🧠 GenAI Initiatives

This is hot right now. Show you’re aware of the trends:

• Vector databases: Store embeddings for semantic search (Pinecone, Weaviate, Chroma)

• RAG (Retrieval Augmented Generation): Combine LLMs with your data

• LLM integration: How do your data pipelines feed GenAI apps?

Real-World Examples to Share

When you mention ML/AI, ground it in real experience:

✅ “In my last role, we built a feature store to support the fraud detection team. We reduced their feature engineering time from weeks to days.”

✅ “I worked with the ML team to create a data pipeline that refreshed model training data daily, which improved model performance by 15%.”

✅ “We implemented a RAG system that allowed our customer support team to query internal docs using natural language.”

✅ “I helped set up MLflow for model versioning and tracking, which gave us reproducibility and audit trails.”

How to Position This in the Interview

• Don’t go deep unless the interviewer asks

• Show collaborative mindset: “supporting the ML team,” “enabling data scientists”

• Keep it high-level architecture, not deep ML theory

• Focus on the data engineering challenges (freshness, quality, scale)

💡 Pro tip: If the job description mentions ML or AI, definitely bring this up. If not, mention it briefly to show breadth, then move on.

6️⃣ Tools Layer: The Professional Touch 🛠️

This is a complementary section where you show you’re not just an ETL developer—you’re a complete data engineer who thinks about the full lifecycle.

This layer is cross-cutting; it applies to all the other layers.

Infrastructure as Code (IaC)

Tools: Terraform, CloudFormation, ARM templates, Pulumi

Why mention this:

• Shows you understand infrastructure should be version-controlled

• Demonstrates reproducibility thinking

• Critical for disaster recovery and multi-environment setup

What to say:

“I’d use Terraform to provision all infrastructure—databases, compute clusters, networking, storage. This gives us version control, automated deployments, and the ability to recreate environments on demand.”

What to provision with IaC:

• Data warehouse / database instances

• Storage accounts / buckets

• Compute clusters (Spark, Kubernetes)

• Networking (VPCs, subnets, firewalls)

• IAM roles and permissions

Version Control

Tools: Git (GitHub, GitLab, Bitbucket)

Why mention this:

“All code—SQL, Python, infrastructure—goes into Git. We follow trunk-based development with feature branches and PR reviews.”

What goes in version control:

• ETL/ELT code

• dbt models

• Infrastructure definitions

• Documentation

• Data quality tests

CI/CD Pipelines

Tools: GitHub Actions, GitLab CI, Jenkins, Azure DevOps

Why mention this:

Shows you think about automation and deployment safety.

What to include in CI/CD:

• Automated testing (unit tests, integration tests, data quality tests)

• Code linting and formatting

• Deployment to dev → staging → production

• Automated rollback on failure

• Notifications to team on failures

What to say:

“Every PR triggers automated tests. Once merged, changes deploy to dev automatically, then staging with approval, then production. This ensures we catch issues early and deploy safely.”

Data Catalog

Tools: DataHub, Collibra, Alation, AWS Glue Catalog, Azure Purview

Why mention this:

Shows you care about discoverability and documentation.

Key features:

• Metadata management: What data exists and where

• Data lineage: How data flows through pipelines

• Business glossary: What do the metrics mean?

• Discovery: Help users find relevant datasets

• Access control: Who can see what

What to say:

“We’d implement a data catalog so business users can discover datasets, understand lineage, and know who to contact with questions. This reduces tribal knowledge and accelerates analytics.”

Monitoring, Alerts & On-Call 🚨

Tools: Datadog, PagerDuty, Grafana, CloudWatch, Azure Monitor

Why mention this:

Shows you understand data engineering is an operational discipline, not just building pipelines.

What to monitor:

• Data freshness: Is data arriving on schedule?

• Data quality: Are validation checks passing?

• Pipeline failures: Which jobs failed and why?

• Resource utilization: Are we over/under-provisioned?

• Cost: Are we staying within budget?

• SLA compliance: Are we meeting our commitments?

What to say:

“I’d set up monitoring for data freshness, quality, and pipeline health. We’d have alerts that page on-call engineers for critical failures, but also slack notifications for warnings. The key is avoiding alert fatigue—only page for things that need immediate action.”

On-call practices:

• Rotation schedule (don’t burn people out)

• Runbooks for common issues

• Post-mortems for major incidents

• Continuous improvement of alert quality

Data Quality Framework

Tools: Great Expectations, dbt tests, Soda, custom validation

Pro tip: dbt can handle all these tests out of the box!

Why mention this:

Data quality is one of the biggest pain points in data engineering. Show you take it seriously.

Types of tests:

• Schema tests: Expected columns present and correct type

• Null checks: Required fields aren’t null

• Uniqueness: Primary keys are actually unique

• Referential integrity: Foreign keys are valid

• Range checks: Values within expected bounds

• Custom business logic: Revenue shouldn’t be negative, etc.

Where to test:

• At ingestion (catch bad data early)

• After transformation (validate business logic)

• Before pushing to production (gate deployments)

What to say:

“I’d implement data quality checks at every stage using dbt tests and Great Expectations. Failed tests block deployment and trigger alerts. We’d track data quality metrics over time to identify degradation.”

FinOps (Cost Management) 💰

Why mention this:

Shows you understand data platforms can get expensive, and cost awareness is part of your job.

What to cover:

• Cost monitoring: Track spending by project/team/environment

• Resource tagging: Tag all resources for chargeback/showback

• Query optimization: Tune expensive queries

• Storage lifecycle policies: Move old data to cheaper tiers

• Right-sizing: Don’t over-provision compute

• Spot/preemptible instances: Use for non-critical workloads

What to say:

“I’d implement tagging for all resources so we can track costs by team. We’d set up alerts for unexpected spend spikes, and regularly review expensive queries and storage to optimize costs.”

Common cost optimizations:

• Partition tables by date (scan less data)

• Cluster tables by commonly filtered columns

• Use materialized views for expensive aggregations

• Compress data (Parquet, ORC, Delta)

• Auto-pause warehouses when not in use

• Use storage tiers (hot, cool, archive)

🎓 Interview Strategy: How to Actually Use This Guide

Before You Start Drawing

1. Clarify Requirements (5 minutes)

• Don’t jump straight into architecture. Ask questions first:

• What’s the core business use case?

• What’s the data volume?

• What are the latency requirements?

• What’s the budget? (Build vs buy)

• What cloud platform?

• What’s already in place?

• Who are the end users?

2. State Your Assumptions

“Based on what you’ve told me, I’m assuming...”

• Daily batch processing will meet requirements

• We’re starting from scratch (no legacy systems)

• We have a team of 5 data engineers

• Budget is moderate (not unlimited, not shoestring)

This shows structured thinking and gives the interviewer a chance to correct you.

While You’re Designing

1. Go Breadth-First, Then Depth

Cover all six layers at a high level first (10-15 minutes). Then ask: “Where would you like me to go deeper?”

This shows you can see the whole system while also knowing when to dive into details.

2. Think Out Loud

“I’m considering two approaches here... Option A would be simpler but less scalable. Option B would handle growth better but adds complexity. Given the requirements, I’d lean toward Option A because...”

This is exactly what they want to hear. There are rarely “right” answers—only trade-offs.

3. Draw as You Talk

Your diagram should evolve as you narrate. Don’t draw everything silently then explain.

“So first, we have these source systems... they feed into... which lands data here... then we transform using...”

Questions to Expect (Be Ready)

Scalability:

• “What if data volume increases 10x?”

• “How does this handle peak load?”

Failure Scenarios:

• “What happens if the source API goes down?”

• “How do you recover from a failed pipeline?”

Security:

• “How do you handle PII/sensitive data?”

• “What about access control?”

Operations:

• “How do you deploy changes?”

• “How do you monitor this?”

• “Who’s on-call when it breaks?”

Trade-offs:

• “Why did you choose X over Y?”

• “What are the downsides of your approach?”

Do’s ✅

• Ask clarifying questions before diving in

• Explain your reasoning for every choice

• Acknowledge trade-offs (nothing is perfect)

• Mention multiple options even if you recommend one

• Show production thinking (monitoring, costs, operations)

• Adapt based on feedback (don’t be rigid)

• Use real examples from your experience

• Keep track of time (breadth first, then depth)

Don’ts ❌

• Don’t assume requirements - ask!

• Don’t jump to a specific tool without justifying

• Don’t over-engineer - start simple, add complexity only when needed

• Don’t ignore operational concerns - it’s not done when the code works

• Don’t claim there’s one right answer - it’s all trade-offs

• Don’t be silent - think out loud

• Don’t go too deep too early - cover breadth first

🚀 Your Action Plan

Now that you have the framework, here’s how to prepare:

📝 Week 1: Internalize the Framework

• Review the six layers until you can draw them from memory

• Practice explaining each layer in 2-3 sentences

• Write down your own experiences for each layer

🎨 Week 2: Practice Drawing

• Pick 3-5 different scenarios (e-commerce, fintech, e-commerce, etc.)

• Draw the architecture for each from scratch

• Time yourself (aim for 30 minutes for a full design)

🗣️ Week 3: Mock Interviews

• Find a peer or mentor to practice with

• Record yourself and watch it back

• Get feedback on communication, not just technical accuracy

📊 Week 4: Deep Dive Your Weak Areas

• Streaming? Read up on Kafka and Spark Streaming

• Data modeling? Study dimensional modeling and normalization

• ML systems? Learn about feature stores and MLOps

🔄 Continuous Improvement

• After every real interview, write down what you struggled with

• Update your own mental model

• Keep learning from real systems you build

🎁 Bonus: Common Scenarios to Practice

Want to test your skills? Try designing these systems:

📱 Scenario 1: E-commerce Analytics

Requirements: Track user clickstream, orders, inventory. BI team needs daily dashboards. Marketing wants near-real-time campaign performance.

💳 Scenario 2: Fraud Detection

Requirements: Credit card transactions, need to flag suspicious activity within seconds. Also need historical analysis for model training.

🏥 Scenario 3: Healthcare Data Platform

Requirements: Patient records from multiple hospitals, strict compliance (HIPAA), clinical researchers need de-identified data.

🎮 Scenario 4: Gaming Analytics

Requirements: Player events, millions of users, need to detect cheating in real-time, also power personalized recommendations.

🚗 Scenario 5: IoT Fleet Management

Requirements: GPS and telemetry from 10,000 vehicles, predict maintenance needs, optimize routes, provide real-time dashboard for dispatchers.

For each scenario:

1. Draw the architecture

2. Justify your choices

3. Identify the biggest technical challenges

4. Estimate rough costs

5. Plan for failure scenarios

📚 Additional Resources

Books:

• Designing Data-Intensive Applications by Martin Kleppmann

• Fundamentals of Data Engineering by Joe Reis and Matt Housley

• The Data Warehouse Toolkit by Ralph Kimball

Helpful articles:

In case you are ready to learn new skills and tools and more or less defined your study roadmap, the next step would be to decide where to practice.

In case, you aren’t please check the road map to get an idea of where are you and what would be your next step.

For sure, you have an absolutely free option to leverage your laptop and install different kinds of free software such as databases, data tools, coding frameworks and so on. However, it would be far from ideal. Far from real-world use cases.

I hope it is crystal clear that you won’t get a job with localhost projects and even if you get one you will struggle in the organization because they run things differently.

So, the best path forward for you work with the most popular vendors. Luckily for us all of them are providing a Trial (Free) period that will work perfectly for your goals.

Let’s start from the baseline - public cloud providers - AWS, Azure, and GCP. You don’t need to learn all of them. Choose one, that you think is the most popular in your area and start building on top of it.

Let’s review what they give us:

Amazon Web Services (AWS)

• Free Tier/Trial: AWS offers a comprehensive Free Tier with 12 months of free usage on many services for new customers (e.g. up to 750 hours/month of a t2.micro VM) (Free Cloud Computing Services - AWS Free Tier). In addition, certain services are Always Free (within usage limits) even beyond the first year (Free Cloud Computing Services - AWS Free Tier). No upfront payment is required, and if usage exceeds free limits, standard pay-as-you-go rates apply (Free Cloud Computing Services - AWS Free Tier).

• Student Program: AWS Educate provides students with free cloud learning resources and credits. Students can access the AWS Educate portal with no AWS account needed, and historically received $35 in AWS credits (up to $100 if their institution is part of the program) (AWS Educate – Credits, Training, Content, and Collaboration for Students & Educators | AWS News Blog). This program offers free online labs and training to help students develop cloud skills.

Microsoft Azure

• Free Tier/Trial: Microsoft Azure’s free account includes a $200 credit valid for 30 days to try any Azure service (Create Your Azure Free Account Or Pay As You Go | Microsoft Azure). It also provides 12 months of free popular services for new users (e.g. 750 hours of Linux/Windows VM, databases, storage) and over 65 services always free within certain limits (Create Your Azure Free Account Or Pay As You Go | Microsoft Azure) (Create Your Azure Free Account Or Pay As You Go | Microsoft Azure). No credit card is needed for the Azure free account sign-up (unless opting to upgrade after the trial).

• Student Program: Azure for Students gives eligible students $100 in Azure credits valid for 12 months, no credit card required (Azure for Students – Free Account Credit | Microsoft Azure). Students can renew this offer each year they remain eligible (i.e. reapply after 12 months for another $100) (Azure for Students – Free Account Credit | Microsoft Azure). This student account includes the same 12 months of free services and always-free products, enabling learners to experiment on Azure at no cost.

Google Cloud Platform (GCP)

• Free Trial: Google Cloud’s Free Program provides $300 in credits for new customers, usable over a 90-day period (Free Trial and Free Tier Services and Products - Google Cloud). During this trial, you can use any Google Cloud services up to the credit limit without charges. You won’t be billed unless you upgrade to a paid account after the credits or time are exhausted.

• Always-Free Tier: Google Cloud also has an Always Free tier – a set of 20+ products with free monthly usage quotas for all users (Free cloud features and trial offer | Google Cloud Free Program). For example, BigQuery allows up to 1 TB of queries and 10 GB of storage free per month (Get started with the Google Trends dataset), and services like Cloud Functions, Compute Engine, and Cloud Storage have free quotas. This lets users continue using GCP in a limited capacity even after the trial (no expiration for these free usage limits).

• Student Programs: Google Cloud offers educational resources and credits through its academic programs. While there isn’t a single unified “student account” like Azure’s, Google’s Cloud for Education initiatives let faculty apply for free credits to use in courses and provide students access to hands-on labs and quests (e.g. via Google Cloud Skills Boost) (Teaching Credits | Google for Education). Students can earn badges and practice with real cloud resources at no cost through these programs (Google Cloud for Education - Students). (Additionally, Google Cloud has partnerships and competitions that grant free credits to students, though these vary by region and time.)

Public cloud skills are vital now. I would think about them in the same was as Microsoft offices in 90s-2000s. So, think about this as just a day to day environment for you data experiments.

The next big thing is Data Platforms. You just need to focus on the leaders - Databricks and Snowflake. Pick one and dive deep.

Let’s check their trial options.

Snowflake

• Free Trial: Snowflake (cloud data warehouse) offers a 30-day free trial which includes $400 worth of usage credits for you to run queries, load data, and test the platform (Snowflake Trial). No credit card is required to start; you just need a valid email to sign up (Trial accounts | Snowflake Documentation). The trial provides full access to Snowflake’s features (standard and enterprise editions) during that period. If you reach the $400 credit limit or 30 days pass, the account will be suspended unless upgraded (Trial accounts | Snowflake Documentation).

• Student Program: Snowflake has an Academic Program called Snowflake for Academia. Through this program, students and educators receive free access to Snowflake’s platform and training materials (learn.snowflake.com). This includes hands-on workshops, industry-recognized Snowflake certifications prep, and a curriculum for instructors. The goal is to help students build data and AI skills on Snowflake at no cost, making them more marketable in data-driven careers (learn.snowflake.com). (Students typically join via their institutions or through Snowflake’s educational initiatives.)

There is a hidden gem on Coursera . These free trainings will give you 6 months trial!!!

Databricks

• Free Trial & Community Edition: Databricks provides a couple of free options. First, there’s a 14-day free trial of the full Databricks platform (on AWS, Azure, or GCP) with premium features for data engineering and ML, so you can test all functionalities (Databricks Community Edition: A Beginner’s Guide). Beyond that, Databricks offers a Community Edition which is an always-free version of Databricks. The Community Edition gives you access to a small micro-cluster (15 GB memory), a cluster manager, and the notebook environment indefinitely (Databricks Community Edition: A Beginner’s Guide) (Databricks Community Edition: A Beginner’s Guide). Your Community account doesn’t expire, making it ideal for learning Spark and trying small-scale projects. (It has limitations in computing power and features, but supports Apache Spark jobs and basic notebooks for free.)

• Student Program: Databricks supports academia through the Databricks University Alliance. This program provides free resources to instructors and students, such as sample course materials, notebooks, and even Azure/AWS credits for classroom use. Educators can get access to Databricks environments to teach, and students get hands-on experience with the Databricks platform (Databricks University Alliance). While not a separate “student edition” of the product, this alliance offers complimentary access to Databricks for educational purposes, helping students learn Spark and big data tools on a real platform.

Let’s check the most popular BI tools:

Looker

• Free Trial: Looker (a business intelligence platform now part of Google Cloud) offers a 30-day free trial for businesses interested in the platform (Free Trial and Free Tier Services and Products - Google Cloud). Since Looker is typically a browser-based SaaS tool, the trial is often arranged by Google’s sales team and allows full use of Looker’s analytics, visualization, and data modelling features on your own data. There are no official forever-free tiers of Looker beyond this limited trial.

• Student Program: Looker does not have a dedicated student license program published. It’s an enterprise-oriented tool usually sold to organizations. Students who want to learn Looker may access it via university programs or sandbox environments if provided by Google Cloud academic initiatives, but there isn’t an official free student edition of Looker itself.

Tableau

• Free Trial: Tableau (a popular analytics and visualization tool) provides a 14-day free trial of its flagship products (Tableau Desktop: Start your free 14-day trial). You can download Tableau Desktop for a full-feature trial or use Tableau Cloud in trial mode. During the trial, you have the same capabilities as a paid user (connect to data, build dashboards, etc.). After 14 days, the software will prompt for a license or the account will revert unless you purchase or obtain an extension.

• Student Program: Tableau has a robust academic program. Tableau for Students offers free one-year licenses for Tableau Desktop and Tableau Prep to students at accredited institutions (Tableau for Students). Students can renew these each year while enrolled. This means as a student you can use the full Tableau Desktop (not just the trial) without charge. The Academic Programs also include e-learning resources and community forums to help students learn Tableau. (Recent updates indicate Tableau is transitioning students to use Tableau Desktop Public Edition for free, but core functionality remains accessible.)

Sigma Computing

• Free Trial: Sigma (Sigma Computing’s analytics platform) offers a 14-day free trial of its cloud BI tool (Does Sigma have a a developer account with extended trial periods? - Forums - Sigma Community). This trial provides full access to Sigma’s spreadsheet-like analytics interface on your cloud data warehouse, with no credit card required to start. Within those 14 days, users can explore data, create dashboards, and experience Sigma’s real-time cloud analytics capabilities. After the trial, continued use requires a paid subscription (Sigma does not have a long-term free tier for its full product).

Microsoft Power BI

• Free Version: Power BI has a free offering via Power BI Desktop and the Power BI Service with a free license. Power BI Desktop (the Windows application for building reports) is free to download and use locally. With a free Power BI license, users can also publish reports to the Power BI cloud for their own use. They can connect to data, create dashboards, and use the Power BI service as long as it's for personal or development use (Power BI service features by license type - Microsoft Learn). The main limitation is that Power BI Free users cannot share dashboards privately with others – sharing and collaboration features require a Pro license (Power BI service features by license type - Microsoft Learn) (Share Report with Coworkers without Pro License - Power BI forums).

• Pro Trial: If you want to try full sharing and collaboration capabilities, Microsoft offers a Power BI Pro free trial (typically 60 days) for new users. This trial lets you test all Pro features – such as sharing reports with colleagues, app workspaces, email subscriptions, etc. – without payment. The official Power BI site invites users to “try Power BI free” which essentially activates the Pro trial within the Microsoft Fabric platform (Power BI: Pricing Plan | Microsoft Power Platform). After the 60-day trial, you would need to buy a Pro license (or have it assigned by your organization) to continue those capabilities.

• Student Access: There isn’t a standalone “Power BI Student” edition, but students often get access through Office 365 Education. Microsoft provides Office 365 A1 licenses free to educational institutions, which include Power BI Free usage. In practice, this means a student with a school-provided Microsoft 365 account can use Power BI to create reports (and if the school has Power BI Pro or Premium capacity, they might be able to share within the school). Additionally, Microsoft offers academic pricing for Power BI Pro (and now Power BI in Fabric) – schools and students can get licenses at a discounted rate (Power BI: Pricing Plan | Microsoft Power Platform). In short, students should check if their university provides Power BI as part of Office 365; if not, they can still use the Desktop and the Pro trial on their own.

And finally, a couple of ETL vendors:

Fivetran

• Free Trial: Fivetran (a managed data pipeline/ETL service) provides a 14-day free trial on its platform (Free Plan). During the trial, you can use Fivetran’s Enterprise Plan features – set up connectors from sources to destinations, and move data – without being charged. Fivetran even allows each new connector you create during the trial to run free for 14 days, so you can stagger testing multiple data sources (New Connection Free Use Period - Fivetran) (Start a free trial with any Fivetran connector for 14 days | Blog). No credit card is needed to start the trial, and you won’t incur costs until you decide to upgrade after the 14-day period.

• Free Tier (Ongoing): Uniquely, Fivetran also offers a Free Plan that has no time limit. After the trial, if your data volumes are small, you can stay on the free tier. The Free Plan allows up to 500,000 rows of data (Monthly Active Rows) and 5,000 monthly active schema changes for free, using Fivetran’s standard connectors (Free Plan). This is great for individual projects or learning, as you can continue running a couple of low-volume connectors indefinitely at no cost. (If you exceed 500k rows in a month, Fivetran will prompt an upgrade and eventually freeze the pipeline until you upgrade or the month resets (Free Plan).)

Matillion

• Free Trial: Matillion (a cloud-native data integration and transformation tool) offers a 14-day free trial for its Matillion Data Productivity Cloud (Free trial - Matillion Docs). When you sign up, you receive 500 free Matillion credits to use within those 14 days, which allows you to run Matillion ETL jobs and try all features. The trial is full Enterprise Edition access, meaning no feature restrictions during that period (Free trial - Matillion Docs). No credit card is initially required; if you don’t upgrade after 14 days (or once the 500 credits are used), the trial ends unless you switch to a paid plan.

Pet Projects Ideas

The tools above will cover the core skills and you can use these tools for free and build great skills. However, the main strength is coming from “pet projects”, i.e. the projects you can build yourself end to end close to real-world use cases.

At Surfalytics, we have lots of pet projects for BI and Data Engineering and you can review them in our GitHub - https://github.com/surfalytics/data-projects

Other helpful resources for our Data Career

1. Free Data Analytics and Data Engineering Fundamentals course

   1. Module 0: Git, CLI, Docker, IDE

   2. Module 1: What is Analytics? How do data teams work? What are data roles and what do they do?

   3. Module 2: Databases and SQL fundamentals

   4. Module 3: BI and Data Viz fundamentals

2. Records of real Data Engineering interviews

3. Record of SQL Mock Interviews with Surfalytics members

4. Blog post about fixing your data Resume
   

Amazon is known for its intense data-driven culture, and one of its secret weapons is the Weekly Business Review (WBR). Imagine the entire leadership team huddled every week, poring over a ton of metrics – that’s exactly what happens at Amazon. In fact, Amazon’s executives review 400-500 metrics in a single 60-minute WBR meeting, and they’ve been doing this every week since the early 2000s.

Why go to such lengths? Because WBRs help leaders catch trends, spot issues, and make informed decisions faster than you can say “Alexa.” It’s not just another meeting; it’s a powerful habit that keeps the company focused, accountable, and nimble in decision-making. If that sounds daunting, don’t worry – we’re going to break down how you can apply a WBR framework to your business (with fewer than 500 metrics, we promise 😉).

Introduction

Amazon is known for its intense data-driven culture, and one of its secret weapons is the Weekly Business Review (WBR). Imagine the entire leadership team huddled every week, poring over a ton of metrics – that’s exactly what happens at Amazon. In fact, Amazon’s executives review 400-500 metrics in a single 60-minute WBR meeting, and they’ve been doing this every week since the early 2000s.

Why go to such lengths? Because WBRs help leaders catch trends, spot issues, and make informed decisions faster than you can say “Alexa.” It’s not just another meeting; it’s a powerful habit that keeps the company focused, accountable, and nimble in decision-making. If that sounds daunting, don’t worry – we’re going to break down how you can apply a WBR framework to your business (with fewer than 500 metrics, we promise 😉).

WBR Explained: What Exactly Is a Weekly Business Review?

A Weekly Business Review (WBR) is a regular, structured review of your business’s performance that happens every week (yes, weekly – consistency is key!). In a WBR, you and your team look at a set of key metrics to understand how the business is doing and why. Amazon popularized this practice as a cornerstone of its management process. Every WBR meeting at Amazon follows a consistent format: a huge deck of charts and numbers covering everything from customer experience to sales to operations, presented in the same layout week after week (How Amazon Uses Input Metrics).

This routine might sound rigid, but it’s incredibly effective. By keeping the format identical each week, Amazon’s leaders can quickly spot anomalies or trends – it’s like flipping through a familiar photo album and instantly noticing what changed since last time.

So why does Amazon take WBRs so seriously? Because weekly reviews drive action. They answer three big questions:

What did customers experience last week? How did the business perform? Are we on track to hit our goals?

In practice, the WBR surfaces unfiltered facts about what’s happening in the business and where it’s heading. If something’s off – say, a spike in customer complaints or a dip in sales – it gets noticed immediately. The discussion then focuses on why it happened and what to do about it. Think of the WBR as an early-warning system and a progress report. Done right, it highlights problems and opportunities, so you can fix issues or double down on a success long before they grow big. No wonder Amazon’s teams treat WBR prep as sacred – it’s their ritual for staying tuned in to the business. And the result? More data-driven decisions and fewer “I didn’t see that coming” surprises 😅.

Let’s check out a real example of WBR from Amazon where Dmitry spent 5 years:

Yes, it was done in Excel and probably still running by Excel and my teams. It took 1-2 days to prepare WBR across all team in the org. Automation with Tableau was a game changer. Any automation is a game changer.

Input vs. Output Metrics: Focusing on What You Can Control

One of the secret sauces in Amazon’s WBR is how they distinguish between output metrics and input metrics. So, what’s the difference? In simple terms: output metrics are the results (the outcomes you want), and input metrics are the drivers (the things you do to influence that outcome. Amazon loves input metrics – and for good reason.

• Output Metrics are like the scoreboard. They tell you how you’re doing, but you can’t change them directly. For example, in business, this could be total revenue, number of customers, or profit – or even your website’s conversion rate. In life, an output metric might be your weight or exam score. These metrics are “lagging indicators” – they lag behind the work that causes them. If your sales or conversion rate dropped last week, that’s useful to know, but it’s just the result. Yelling “increase!” at an output metric usually doesn’t help (if only it were that easy! 😜).

• Input Metrics are like the levers and dials you can turn to affect the outputs. They are controllable actions or processes that drive those results. Think of things like a number of new products added to your catalog, website loading speed, marketing spend, customer service response time, etc. These are typically “leading indicators” – they lead to outcomes. The beauty of input metrics is that when one is off track, you know where to intervene. They give you a clear idea of what to fix or improve. For instance, if “website uptime” (an input) fell, you can bet it’s impacting “orders placed” (an output), and you know exactly which tech team to poke.

Amazon’s philosophy is to obsess over input metrics. Why? Because focusing on controllable inputs is how you eventually get the outputs you want. A famous anecdote drives this home: Jeff Bezos once met a CEO who was giddy that his company’s stock price went up by 30 cents that day – a classic output metric. Bezos basically shrugged and noted the CEO had done nothing to cause that stock rise.

The lesson? Don’t pat yourself on the back for outputs you don’t directly control. Instead, zero in on the inputs you can control (like improving customer experience, product selection, or co which in turn drive those outputs. It’s like trying to lose weight – you can’t just tell the scale to change (output), but you can exercise and eat right (inputs) and let those efforts reflect on the scale over time. By managing the right inputs, Amazon ensures the outputs (revenue, market share, customer satisfaction, you name it) take care of themselves. As one Amazon veteran put it, output metrics are important, but input metrics are what give you “clarity on the levers you can pull” to improve those outcomes. In short: control the inputs, and the outputs will follow.

eCommerce Example: WBR in Action at “Surfalytics” 🏄‍♂️

Let’s bring this to life with a practical example. Meet Surfalytics,

https://shop.surfalytics.com/

our real eCommerce store that sells surfing gear and beach accessories. How can Surfalytics use a WBR and input/output metrics to ride the wave to success? 🌊

Imagine every Monday morning, the Surfalytics team holds a Weekly Business Review meeting. They look at a simple one-page report of last week’s key stats. Here’s how they might break it down and use those metrics:

• Conversion Rate (Output) – Last week, Surfalytics’ conversion rate (the percentage of visitors who bought something) was, say, 2.5%, down from 3% the week before. That’s a red flag 📉. In the WBR, the team asks “Why did it drop?” They dive into input metrics behind conversion. It turns out the website load time increased by 1 second last week (perhaps due to a fancy new video on the homepage), and a couple of popular products were out of stock. Slower pages and missing products likely frustrated some shoppers, causing fewer checkouts. Aha! Now the team has actionable steps: optimize the site speed and urgently restock those popular items. By tackling these inputs, they expect the conversion rate to bounce back rather than just hoping numbers magically improve.

• Traffic (Input) – Next, they review website traffic. Good news: traffic is up 15% week-over-week, thanks to a new Google Ads campaign 🎉. Traffic is an input metric because Surfalytics can control it (by adjusting ad spend, SEO, content marketing, etc.). Higher traffic usually leads to higher sales if other inputs (like site experience and product availability) are solid. The team notes that the campaign worked well. If traffic had been down, that would have signalled the need for action (maybe the marketing team paused ads or some referral source dried up, which they’d address for next week). By tracking it weekly, they’re never in the dark about their customer visits.

• Product Availability (Input) – Surfalytics tracks the in-stock rate of its top products. Last week, it was only 80%, meaning 20% of high-demand items were unavailable. That’s likely a big factor in lost sales – you can’t sell what you don’t have, and customers probably went “out of stock… ugh” and left. In the WBR, this metric prompts a plan: the inventory team will work with suppliers to raise that in-stock rate to 95%+. Product availability is a controllable input – the team can introduce better inventory management or reorder faster. Improving this input will directly boost output metrics like sales and customer satisfaction (fewer folks encountering out-of-stock messages). Lesson learned: always keep your best-sellers on the (virtual) shelves!

• Operational Efficiency (Input) – The Surfalytics crew also checks an operational metric: say, order fulfillment time (how fast orders are shipped out). They see it slipped from an average of 1 day to 2 days last week. Maybe the warehouse was understaffed or a process snag slowed things down. While customers haven’t started screaming yet, a slower ship time could hurt the customer satisfaction score or lead to higher refund requests (output metrics) if it continues. Because WBR highlighted this trend early, the operations manager can act now – perhaps put an extra team on duty or streamline packing for the coming week. By tightening this up (input), they keep the customer experience strong and prevent future issues. In essence, the WBR helps them dive deep into each part of the business, from web analytics to supply chain and ensures every team knows where to focus their efforts before small problems become big ones.

Through this weekly routine, Surfalytics creates a continuous improvement loop. Each week they celebrate wins (🤝 high-five for that traffic boost) and confront challenges head-on. Over time, this leads to better and better performance. The marketing folks see how their campaigns translate to sales, the product team sees how selection and stock levels matter, and the operations folks see the impact of speedy delivery – it’s all connected. The WBR turns vague goals like “increase revenue” into specific action plans: increase revenue -> improve conversion -> speed up the site, keep products in stock, etc. It’s business analytics made actionable and even a bit fun (with coffee and donuts in the meeting, of course!).

WBR in Tableau

Let’s create a WBR dashboard in Tableau! We’ll use the Sample Superstore Dataset for this project. If you prefer, you can simply download the workbook and reverse-engineer it. The final solution is available here.

To save time on dashboard layout, I’ll be using my free **Dashboard Template.** You can download it and start building your charts right away.

I recommend using the 3x2x1 layout and completely deleting the left column for a cleaner design.

Weekly Line Chart

This is Line Chart that shows Weekly Sales values, its Previous Period Sales (in this case previous Week) and its’ Previous Year Sales.

1. Build the Chart Axes

1.1 Drag Order Date to the Columns shelf. Change its granularity to Week by clicking on the dropdown menu for Order Date and selecting Week.

1.2 Add Sales to the Rows shelf

2. Create Calculations

You will need calculated fields for:

• Previous Week Sales. Let’s create a calculation called Sales Previous Period because we’ll use it in Previous Monthly Sales as well..

//**Sales Previous Period**
LOOKUP(ZN(SUM([Sales])), -1)
//LOOKUP function shifts the focus from the current row to one row backward 

• Sales Previous Year Week. This one will be separate from the monthly one because we’ll calculate not -1 row, but -52, meaning 52 weeks ago.

//**Sales Previous Year Week**
LOOKUP(ZN(SUM([Sales])), -52)
//LOOKUP function shifts the focus from the current row to 52 rows backward 

3. Combine Measures on the chart

3.1 Drag Measure Values to the Rows shelf.

Filter by Measure Names to leave only Sales Previous Periodand ****Sales Previous Year Week

3.2 Create a Dual axis between SUM(Sales) and Measure Values by right-clicking on Measure values → Dual axis. Synchronize axes by right-clicking on them → Synchronize axes.

4. Filter the chart

One of the tricks here is to filter for the last 6 weeks while still keeping all the LOOKUP calculations visible. Due to the order of operations, we can’t use the usual dimension date filter. Instead, we need to use a Table Calculation filter. For more details, refer to Tableau's Order of Operations.

This is why we create a filter calculation:

//**Filter Rank date (week)**
RANK(MAX(DATETRUNC('week',[Order Date])))
//Applies RANK to dates in the chart

Then, use this calculation to keep only the last 6 ranked weeks.

5. Customize the Marks

For SUM(Sales) set up the blue colour. For Measure Names set up light grey for Sales Previous Period and dark grey for Sales Previous Year Week. Choose a dashed line type for them.

6. Formatting

• Hide right axis by right click → Un-click Show header.

• Remove date axis title.

• Remove Column and Row dividers. Remove Grid Lines.

• Use grey for axes text.

More about Formatting in Tableau: https://help.tableau.com/current/pro/desktop/en-us/formatting_worksheet.htm

• Add legend via the Sheet title – fact -- previous week -- previous year

Monthly Line Chart

This is Line Chart that shows Monthly Sales values, its Previous Period Sales (in this case previous Month) and its the Previous Year Sales.

For a monthly line chart, the logic is the same.

The only differences are:

• Sales Previous Year Month: This calculation helps find the sales from the same month in the previous year.

//**Sales Previous Year Month**
LOOKUP(ZN(SUM([Sales])), -12)
//LOOKUP function shifts the focus from the current row to 12 rows backward (yar ago)

• Filtering the chart to display only the last 6 months.

//**Filter Rank date (month)**
RANK(MAX(DATETRUNC('month',[Order Date])))
//Applies RANK to dates in the chart

Now you can place both of these charts on the dashboard.

Weekly KPI Card

This is KPI card that shows Last Week Sales values, its Period over Period % change (in this case % WoW) and Year over Year % Change.

This is not an essential part for the WBR but it can be useful to look at the charts together with KPI cards.

1. Build the KPI structure

1.1 Drag Order Date to the Columns shelf. Change its granularity to Week by clicking on the dropdown menu for Order Date and selecting Week. Make it Discrete.

1.2 Add Sales and WEEK(Order Date) (you can copy it by Cmd/Ctrl+Drag the pill from 1.1) to the Text in the Marks shelf.

Go to the text and add text ‘Sales’ on top of the SUM(Sales)

Apply formatting:

• black color and size 22 for SUM(Sales)

• black color and size 14 for ‘Sales’ text

• grey color and size 10 for WEEK(Order Date)

• Apply Custom Date Formatting to WEEK(Order Date): dd mmm. More about Custom Date Formats

• left-align all text

At the end, you’ll get something like that

2. Create Calculations for PoP

We’ll reuse some of the previously calculated fields, like Sales Previous Period and Sales Previous Year Week

• Firstly, let’s create the one to calculate % difference between periods – Sales % PoP.

//**Sales % PoP**
(ZN(SUM([Sales])) - [Sales Previous Period]) / [Sales Previous Period]

• Now we need two more calculations to create ↑↓ % for the KPI card on the basis of Sales % PoP

//**Sales % Neg**
IIF([Sales % PoP] < 0,
'↓' + STR(ROUND([Sales % PoP]*100,1)) + '%','')
//Here we check if the % PoP was **negative** and if yes we add arrow down ↓ and % result.

//**Sales % Pos**
IIF([Sales % PoP] > 0,
'↑+' + STR(ROUND([Sales % PoP]*100,1)) + '%','')
//Here we check if the % PoP was **positive** and if yes we add arrow up ↑ and % result.

3. Add PoP to the KPI card

Now we can add them to the KPI card design. Just place Sales % Neg and Sales % Pos them in the to the Text in the Marks shelf.

Now the trick would be to apply formatting, Since the Sales % PoP ****can’t be both positive and negative at the same time, we’ll only see Sales % Neg OR Sales % Pos.

So in the text we color Sales % Neg in red-orange and Sales % Pos in blue-violet and add ‘WoW’ at the end.

4. Create Calculations for Week YoY

Now let’s create the rest of the calculations for the KPI card – YoY.

• Firstly, let’s create the one to calculate % difference between years – Sales % WoW YoY

//**Sales % WoW YoY**
(ZN(SUM([Sales])) - [Sales Previous Year Week]) / [Sales Previous Year Week]

• Now we need two more calculations to create ↑↓ % for the KPI card on the basis of Sales % WoW YoY

//**Sales % Neg WoW YoY**
IIF([Sales % WoW YoY] < 0,
'↓' + STR(ROUND([Sales % WoW YoY]*100,1)) + '%','')
//Here we check if the % YoY was **negative** and if yes we add arrow down ↓ and % result.

//**Sales % Pos WoW YoY**
IIF([Sales % WoW YoY] > 0,
'↑+' + STR(ROUND([Sales % WoW YoY]*100,1)) + '%','')
//Here we check if the % YoY was **positive** and if yes we add arrow up ↑ and % result.

5. Add YoY to the KPI card

Do the same as in the 3rd point by adding Sales % Neg WoW YoY and Sales % Pos WoW YoY to the text and formatting it.

You should get smth like that at the end.

6. Filter by last week

To filter by last week we can use Filter Rank date (week) calculation and leave only 1st one.

Monthly KPI Card

This is KPI card that shows Last Month Sales values, its Period over Period % change (in this case % MoM) and Year over Year % Change.

For the monthly card the logic is the same.

But instead of Sales % WoW YoYcalculate Sales % MoM YoY using Sales Previous Year Month

//**Sales % MoM YoY**
(ZN(SUM([Sales])) - [Sales Previous Year Month]) / [Sales Previous Year Month]

And based on it the same calculations for Sales % Neg MoM YoY and Sales % Pos MoM YoY

Filter by Filter Rank date (month) to leave only 1st rank.

Now you can add both KPI cards on top of the Line Charts

Table Weekly

This is the table that shows Weekly Sales values, its Period over Period % change (in this case % WoW) and Year over Year % Change.

1. Drag Order Date to the Columns shelf. Change its granularity to Week by clicking on the dropdown menu for Order Date and selecting Week.

2. Add Segment and Measure Names to Rows**.** Drag Measure Names to the Rows shelf.

3. Add Measure Values to the Text in the Marks.

4. Filter by Measure Names to leave only

• SUM(Sales)

• Sales % PoP

• Sales % WoW YoY

All these calculations were created during the KPI card creations.

5. Change Marks type from Text to Square.

6. Add Measure Values to Color in the Marks.

7. Use Separate Legends for Measure Values.

8. Set up colors.

For Sales % PoP and Sales % WoW YoYchoose diverging colors, I use red orange ##FF7E79 and blue violet #7A81FF

For SUM(Sales) I also use diverging colors but put white #FFFFFF on both side so the cells with SUM(Sales) won’t be colored.

9. Filter by last 6 weeks

To filter by last week we can use Filter Rank date (week) calculation and leave 1-6 ranks.

10. Add worksheet title ‘WEEKLY’

Table Monthly

This is the table that shows Monthly Sales values, its Period over Period % change (in this case % MoM) and Year over Year % Change.

For Table Monthly the logic is the same.

But instead of Sales % WoW YoY use Sales % MoM YoY and Filter Rank date (month) for filtering.

I also removed Last unclosed month from it for use case purposes by using

//Filter Last unclosed month
DATETRUNC('month', [Order Date])<{MAX(DATETRUNC('month', [Order Date]))}

Hide Segment and Measure Names by right-click → unclick Show header.

Table MTD

This is the table that shows MTD – Month to Date Sales values, its Period over Period % change (in this case % MoM) and Year over Year % Change. MTD measures the performance of a metric from the start of the current month up to the present day.

For MTD Table we gonna need a bit more calculations.

1. First let’s calculate current MTD Sales – sales from the start of the current month till the last available day. For example, if today is 20 November 2024, we’ll calculate the SUM of Sales from the 1st of November to 20th of November.

//**MTD Sales**
IIF(
[Order Date] <= {MAX([Order Date])}
AND
YEAR([Order Date]) = {MAX(YEAR([Order Date]))}
AND
MONTH([Order Date]) = MONTH({MAX([Order Date])}),
[Sales],
NULL)

1. Now Previous month MTD Sales – sales for the same period as current MTD but month ago. For example, if today is 20 November 2024, we’ll calculate the SUM of Sales from the 1st of October to 20th of October.

//**Previous month MTD Sales**
IIF(
[Order Date] <= DATEADD('month', -1, {MAX([Order Date])})
AND
YEAR([Order Date]) = YEAR(DATEADD('month', -1, {MAX([Order Date])}))
AND
MONTH([Order Date]) = MONTH(DATEADD('month', -1, {MAX([Order Date])})),
[Sales],
NULL)

1. Now we’ll calculate the % difference between 1. and 2. to get MTD Sales % PoP

//**MTD Sales % PoP**
(SUM([MTD Sales])-SUM([Previous month MTD Sales]))/
SUM([Previous month MTD Sales])

1. Next we’ll need Previous year MTD Sales – sales for the same period as current MTD but year ago. For example, if today is 20 November 2024, we’ll calculate the SUM of Sales from the 1st of November 2023 to 20th of November 2023.

//**Previous year MTD Sales**
IIF(
[Order Date] <= DATEADD('year', -1, {MAX([Order Date])})
AND
YEAR([Order Date]) = {MAX(YEAR([Order Date]))} - 1
AND
MONTH([Order Date]) = MONTH({MAX([Order Date])}),
[Sales],
NULL)

1. And YoY change between current MTD Sales and Previous year MTD Sales

//**MTD Sales % YoY**
(SUM([MTD Sales])-SUM([Previous year MTD Sales]))/
SUM([Previous year MTD Sales])

1. Create the table.

The logic of table creation is the similar to the Tables Weekly/Monthly.

6.1 But to columns put Last Month Date

//**Last Month Date**
{MAX(DATETRUNC('month',[Order Date]))}

6.2 Use MTD Sales, MTD Sales % PoP, MTD Sales % YoY as measures for Rows.

6.3 Hide Segment and Measure Names, apply formatting.

Table QTD

This is the table that shows QTD – Quarter to Date Sales values, its Period over Period % change (in this case % QoQ) and Year over Year % Change. QTD measures the performance of a metric from the start of the current quarter up to the present day.

All the calculations for QTD are similar to MTD but instead of ‘month’ use ‘quarter’.

You can check all of them in the Tableau workbook.

Table YTD

This is the table that shows YTD – Year to Date Sales values, its Period over Period % change (in this case % YoY). YTD measures the performance of a metric from the start of the current year up to the present day.

For the table instead of % PoP we’ll put MIN(NULL) to have the line but not to put values in it.

All the calculations are similar to MTD but use ‘year’ instead of ‘month’.

You can check all of them in the Tableau workbook.

Tables all Together

To create a full table we’ll use all the one we previously created and put them into the horizontal container one by one. Make 0 paddings and apply Fit → ‘Entire view’.

Between these tables you can put Blank objects with grey background 0 paddings and 1pt width.

Sources:

1. Cedric Chin, Commoncog: “The Amazon Weekly Business Review (WBR)” – Describes Amazon’s WBR meeting cadence and its role in Amazon’s data-driven ( The Amazon Weekly Business Review (via Bezos's Shadow) - Commoncog ).

2. Paul Duvall, Mastering Weekly Business Reviews – Highlights how WBR is used as a mechanism to analyze trends and drive account (Mastering Weekly Business Reviews: Insights from Amazon's Iconic WBR) (Mastering Weekly Business Reviews: Insights from Amazon's Iconic WBR)†L56-L59】.

3. Holistics.io Blog: “How Amazon Uses Input Metrics” – Explains Amazon’s focus on controllable input metrics vs. output metrics (How Amazon Uses Input Metrics) (Input Metrics for Business Growth: A Guide to Defining and Measuring Key Drivers).

4. WorkingBackwards.com, “Input Metrics & Weekly Business Review” – Defines input and output metrics (controllable inputs that improve customer experience vs. lagging outputs (Input Metrics for Business Growth: A Guide to Defining and Measuring Key Drivers). Also outlines Amazon’s practice of reviewing metrics weekly, and how this fits into broader monthly/quarterly (Amazon Operating Cadence - Working Backwards).

5. The North (thenorth.io): “Input metrics vs Output metrics” – Provides a simple overview of inputs (actions you control) vs outputs (results you want) (Input metrics vs Output metrics - The North) (Input metrics vs Output metrics - The North).

This is an example of resume feedback from our community experts. The example below is almost the final product after several iterations. The initial version of it was useless. As a result, this particular member of Surfalytics landed the job in NY with no prior experience working in states. It took months but hard work paid off.

And this is the review of this member:

We have a dedicated Discord channel where we practise improving our Resume and avoiding any 🟥 flags.

In this post, I want to highlight low-hanging fruits that might help you save time and find a job.

Match Job Titles

Every time I review someone's resume, I can instantly spot the biggest issue - the title and experience don’t match the job posting.

Assume, you want to apply for the Data Engineer role. You take your favourite resume with the title Data Analyst and apply for Data Engineer.

Obviously, Data Analyst <> Data Engineer. This is 99% rejection. Data analyst and data engineer examples are not the worst use cases.

Some people might have different job titles on their resume

• IT helpdesk

• Front end developer

• Barista

• Couriers

• and etc

It is great that you have a non-relevant experience that might give you valuable skills BUT there is 0 relevance to the job you are applying.

Let’s assume you have this kind of Resume. In high-level summary:

Target title: Data Specialist

Location: Different Country for Job Application

Summary:
Super puper result oriented professional with 1 year of exerience and bla bla

Work Experience:
Data Specialist at Z 01.2023-05.2024
- your duties

Call Center operator at Y 06.2021 - 12.2023
- your duties

Line cook at X
- your duties

You decided to apply for Data Engineer. In my opinion, it is a waste of your time and time of HR to review your application. There are some interactions with word data, but probably that’s it.

If you are applying for a job with title X, but in your resume online title A, B, C. You have had a very low chance of getting even a call from HR.

Are you curious why? The answer is simple: The DATA profession is now a popular topic. There are hundreds of trainings, workshops, MBA and Master's degrees, and thousands of data influencers.

Everyone wants to work from home for high $$$ on their MacBook with Oat Milk latte. Reality is different.

Based on my experience people with MBAs and Masters in Analytics, or even folks with non-relevant experience are suffering in the same way. They all are struggling to find a job. Because they don’t have real skills that matter.

Instead of wasting time and money, you need to try to narrow your focus and define your specialization. Pick 1-2 job roles and make the resume 100% match.

At least, you will get a shot for a screening call with HR.

Let’s tweak a bit our resume:

Target title: Data Engineer

Location: Same country as job application (assume you have work authorisation)

Summary:
- 4 years of Data Engineering
- ....
- ...

Work Experience:
Data Engineer at Z 01.2023-05.2024
- your DATA Engineering duties

Data Engineer at Y 06.2021 - 12.2023
- your DATA Engineering duties

Data Analyst/Engineer at X 01.2020 - 06.2021
- your DATA Engineering duties

We tweaked it to look like a good Data Engineer with 4 years of experience.

Before going crazy and apply all jobs let’s make sure:

• You actually have Data Analytics or Data Engineering experience. If not, you can gain it in 4-6 weeks to prepare for job applications. That’s why at Surfalytics we run almost weekly data engineering and data analytics projects from real-world cases and practice interviews.

• Be ready to fail 5-10 real interviews before you get a job. Every failure is a big success, celebrate it. It is real feedback on your effort.

Summary of your profile

Let’s review the Summary of your resume. Almost every resume I saw has a useless summary. What a waste of time. According to research most recruiters look at the top of your resume:

They also spend ~ 6 seconds per resume. That’s why it is in hour interest to make the Summary insightful and to the point. Remove all useless verbs like result-oriented, experienced and so on.

Example of bad one:

As a Canada-based Data Engineering contractor, Dmitry holds a Master's in Engineering and has 15+ years of international experience in Business Intelligence, Data Warehousing, Big Data, Cloud, and ML. He has built data platforms with cloud and big data technologies and supported machine learning experiments, data science models, and business intelligence reporting. He adeptly manages privacy compliance and data security. He operates via Incorporation and possesses a valid Secret with the Canadian Government.

Improved one:

• 15+ years of building and leading modern Data Engineering Solutions.

• Engineered data pipelines using Amazon Redshift, AWS Glue, EMR, Databricks for processing over 100 TB of data.

• Migrated legacy analytics systems to Snowflake, dbt, Tableau, Looker.

• Managed and created data infrastructure using modern DevOps practices, Infrastructure as a Code, CI/CD.

• Mentored and hired Analytics and Data Engineering teams.

• Built Iceberg Lake House using Amazon Athena, Airbyte for FinTech company.

It is not perfect but I hope you got an idea. It is a place to make a point about your experience.

Geographic market specific

It is an important and obvious rule. You need to have the Work Authorisation in the country to which you are applying. Don’t be naive to apply to Canada and the US from India or Kazakhstan. It won’t work. There are thousands of eligible people in the country itself.

Better find a way to get a work visa officially.

Sometimes, you can try to make a trick like I did in the past. I was in Russia and was searching job in Winnipeg, Canada. Obviously, HR will turn down all my applications with non-Canadian addresses. I made a trick. I’ve added the Winnipeg Address of my friend and used IP telephony to forward it to my Russian number. My goal was to reach the hiring manager.

It worked out, I got an interview with the hiring manager and explain that I was waiting Permanent Residence card. They made an offer and waited for 6 months while my documents were processed.

The gap in your experience

Often you might be in a situation with a gap in the experience:

Target title: Data Engineer

Location: Same country as job application (assume you have work authorisation)

Summary:
- 4 years of Data Engineering
- ....
- ...

Work Experience:
Data Engineer at Z 01.2023-12.2023
- your DATA Engineering duties


Data Analyst/Engineer at X 01.2020 - 06.2021
- your DATA Engineering duties

This gap looks terrible, at least for me. Do you want to risk your lifetime opportunity to miss an interview due to this gap? At Surfalytics we worked towards closing the gap and looking into the opportunity to close the gap. But the best way is to close the gap is: to extend the date to close the gap. This looks solid:

Target title: Data Engineer

Location: Same country as job application (assume you have work authorisation)

Summary:
- 4 years of Data Engineering
- ....
- ...

Work Experience:
Data Engineer at Z 01.2023-12.2023
- your DATA Engineering duties


Data Analyst/Engineer at X 01.2020 - 01.2023 (extend the date
- your DATA Engineering duties

Alternatively, you can extend your last job till the current time.

No one really cares about your past. The most important is your skills. Are you able to perform well? Are you ready to compensate lack of skills with a 60-80-hour working week? Moreover, there are different “Privacy laws” that will protect your personal life and your past;)

There is no easy road to success and hard work will be paid off!

Tools and use cases

Another aspect of your resume - your job duties and tools. For every role, we can define 20% tools and skills that matter the most. In another article, I’ve covered this topic:

For example, you probably want to share your actual experience and duties and best thing you can do something like:

- Excel
- Reports and Data Analysis in excel
- Pivot and cross tables in excel
- etc.

The point is that you will put something that might look like a 100% Data Analytics tool but in real life, it is not true. Even if you are working as an analyst, you probably work with product managers, and data engineers at the organization and you should be aware of their work and tools:

- Building Product usage dashboards in Tableau for Product department to track key product metrics like DAU, cohort analysis
- Developed Self-Service report for Marketing manager to track Ad spent
- Work close with Data Engineering team
- Ensuring quality of busienss metrics in data engineering pipeline using dbt tests and Great Expectations
- Maintain documentation in Data Catalog
- etc

In my example, I’ve added more specific examples. My examples aren’t boring, they are actually cool and I wish to work with this person vs just “cross table pivot in Excel”.

Again, no this kind of experience? Let’s get it. By the way best book to gain knowledge for business metrics is Lean Analytics.

Every Surfalytics member starts their journey from this book 🚀.

Your Portfolio

Another great asset for you as a data professional might be thinking about the portfolio.

There are a couple of cool things you might try:

• Active GitHub account

  

• Personal Blog. But you should choose the dedicated theme. You can have a blog about “all things data”. Specialization is key in blogs, resumes and so on.

• Contribute to open-source products. Even with limited code knowledge you can read and improve Markdown files and documentation.

• Participation in data community or user groups like Surfalytics where you can run the projects and share the knowledge

• Presenting local user groups for top data vendors: Databricks, Snowflake, Microsoft, Tableau etc.

• Presenting at conferences.

Basically, anything data-related that you are passionate about and you make.

Certification and Education

Usually in the resume, we have a dedicated section for Education and Certifications. In most cases for data jobs, skills and experience matter the most. If you have high education, it is great. In case it is not relevant, just hide specialization. It is nice to have something.

If you don’t have anything, just go to Coursera and finish something relevant and call it as your education;)

The certifications are even more fun. There are tons of certs. They have zero value for anyone. Usually, the training for certification preparation like dbt, Tableau, Snowflake, Databricks, Azure, AWS and etc are great. But the passing exam is a different story. By the end of the day if you feel more comfortable with certification, just add this into your resume without passing. But make sure you have the desired skills and knowledge and at least finish the training.

How many copies of resumes do you need?

We are coming to an interesting question. I’ve heard that people are tired of tailoring their resume for each position they apply.

Fortunately, it is useless. Instead, you should prepare several resumes with dedicated specialization:

• Data Analyst Resume

  • Product Analyst Resume

  • Marketing Analyst Resume

  • Finance Analyst

  • DevOps Analyst

• Analytics Engineer

• Data ENgineer

  • Azure Data Engineer

  • GCP Data Engineer

  • AWS Data Engineer

I hope it makes sense for you that you will slightly change the tooling and use cases but the core will email the same.

My example:

PDF vs Doc

Sometimes, I can see a very nice resume built-in PDF. Unfortunately, it might be hard to read and consume by HR or ATS Systems.

Forget about fancy PDF files with bar charts and other nonsense. Let’s use simple Google Docs and simple text. My example:

My example isn’t perfect. But still a great point to start. I would also add more details about BUSINESS VALUE and actual IMPACT on business and data team.

Resume vs Linkedin

You might now wonder what to do if you have several resumes ranging from Data Analyst to Data Engineer.

Sometimes, HR or even hiring managers might quickly check your LinkedIn profile and their expectations that your LinkedIn should match your Resume. The key here is to match:

• Date ranges

• make sure the keyword “data” appears in your title.

For example, instead of Data Analytics, you can write Data at XYZ company. This simple trick will replace a family of data jobs.

Again, the best possible strategy you can get is SPECIALIZATION.

If you are curious about the difference between roles, I recommend you to check the simple data roadmap from 0 to hero data engineer:

Eh, cover letter?

I doubt anyone will check your cover letter. Don’t waste time.

Competitive Advantage

This is what we actually do at Surfalytics with our pet projects, mock interviews and overall support - we are looking for ways to gain a natural competitive advantage that will help you stand out among other candidates, let me share ideas:

• Understanding the Value of Analytics

• Knowing the pain points of Hiring Managers and proactively offering solutions

• Using the keywords specific to the role

• Understanding end-to-end architecture of analytics solutions

• Domain knowledge and key analytics use cases for the industry

• Never apply on LinkedIn. It is useless in 95%, go directly to the company website.

• Networking is a great way of learning about openings in your area. Check all user groups for Snowflake, Power BI, Tableau, Databricis, DBT and other data-related meetups and conferences!

• Try to grow your LinkedIn presence, write 2-3 posts per week. Again, think about a dedicated topic or tool.

Another good advice could be to leverage friends for a direct reference to the company. However, the current market is difficult and there are so many candidates.

The best thing you can do is focus on things you can control and make them look perfect. This will increase your chances of starting a conversation with the company and hiring manager.

References

• Google Doc resume template

• A practical guide to writing FAANG-ready software engineer resumes

• How to Create an Impressive GitHub Profile README

• Resume score

• Career advice for long-term career ““Essential Tips for Career Success based on real story”: Part 1, Part 2, Part 3

• Soft Skills: The software developer's life manual

  • 
    

I also created an online exhibition of Executive Layoff Memes to remind us that the corporate world isn’t family and that you can be disposable in a single day. However, I eventually dropped this since I wasn’t sure about the maintenance cost.

I have 15 years of experience in the industry, with 10 of those years in North America . I am still working as a data engineer. I had some cool experience in Amazon and Microsoft in the past. I spent almost 5 years at Amazon, working in Vancouver, Boston, and Seattle. And several years with Microsoft working with Xbox and Azure teams.

In this article, I want to focus on the interview process, touch on issues of employer loyalty, financial rewards, career growth, and future prospects, and weigh the pros and cons of “working for someone else” versus “working for yourself.”

You might find some of typos. Unfortunately, grammar isn’t my strong suit, unlike navigating interviews, where I feel “right at home.”

Before moving on to companies like Amazon and Microsoft, I’d like to start with some simpler examples.

Getting Started

I originally trained as a design engineer, and I genuinely tried to work in my field (for a full 3 years) in Moscow, Russia.

Before I discovered that an assistant project manager earned $1,500 (back in 2009), while the head of a metalworking department at a space center made less than $700.

Tip 1: If you’re thinking that in the future you’ll land a great job with a good income, and all you need to do is put in X years, complete grad school, take Y courses, and get Z certifications—put those assumptions to the test. You might save time, resources, and money.

Now, I know this as “hypothesis testing.” While working at the factory, I decided to test my own hypothesis. I posted a resume on HeadHunter (HH), claiming several years of experience and a graduate degree, and started applying to positions.

I quickly realized that all the available jobs in the Moscow region offered laughable salaries. That’s when I began to question the future of the engineering industry in Russia. It turns out I wasn’t wrong.

When I was in university, I really wanted to intern in Europe. But at some point, those programs were simply canceled. The next image captures this perfectly.

I had to retrain. Still am, actually. In IT, as soon as you start learning one thing, something new pops up, and what you just learned is already outdated.

Tip 2: Fake it till you make it (FITYM).

Now I know this trendy phrase, but back then, I just thought I needed to embellish my experience a bit. I had to take my factory experience and turn it into a good story—an IT story. FITYM can apply to almost anything. It’s a catch-22: they demand experience, but you don’t have any, and you can’t get any because they won’t hire you without it. FITYM is the solution to that problem.

Even with my FITYM approach, I didn’t have much success—just 2-3 interviews. Then I got offered an internship at a bank in the IT department, specifically in the Business Intelligence role for report development. They asked me just one question: “Do you know what SQL is?” Although it was the first time I’d heard the word, I confidently replied, “Of course I do! I even used SQL in my thesis project.”

Tip 3: Take risks. You only live once, so it’s better to take a chance. As someone once said, “It’s not scary to make mistakes—it’s scary to be a dull nobody.”

And this brings up some questions—a sort of FAQ:

1. What if I lie about my experience, and it turns out I don’t have it, and I get fired in disgrace?

   
   Of course, you need to know your limits. Don’t claim you’re a Java architect if you’ve only just written your first “Hello, world” program. But if you understand how to solve problems in theory, where to find answers, and how to ask the right questions, you’ll manage. Besides, hiring someone is a time-consuming and costly process. Your skills are just part of the picture; it’s also about how well you fit in with the team. In the beginning, you can make up for a lack of experience by putting in extra hours. Work 16-hour days, no weekends—you’ll figure it out.

2. My resume has irrelevant experience.

   
   The first thing we do is rewrite the resume. Add the experience that’s relevant to the employer. I’ve never encountered a situation where recruiters compared your resume with your employment record. Be creative; put in the resume what you need. Only include “job location” and “employment dates.” You can tailor the “job title” and “responsibilities” as needed.

3. Fine, no problem with the resume. But what about the employment record?

   
   HR departments are run by people, too. They’re often swamped, not highly paid, and need time for coffee breaks, social media, and gossip. So they’re unlikely to have time to comb through your employment history with a fine-tooth comb.

4. What about the security department?

   
   Big companies have security departments. For them, you’re just a name, and they might do a background check. This is a bit of a roll of the dice. You could leave a friend’s number, or come to some arrangement. I’ve even had a case where I had to take a lie detector test. But I got through it. The main thing for them is that you don’t have a criminal record.

The first three months are the hardest. The people around you aren’t dumb; they’ll quickly catch on that you’re short on knowledge. But! You’re making an effort—arriving early, leaving late, asking the right questions, and being very proactive. Where are they going to find a better candidate?

Job Change – From Banking to an IT Vendor

After 1.5 years, I’d gained some solid experience, and it was time to hit the market.

Tip 4: The odds of working hard and getting a double salary increase are close to zero. The only real way to raise your salary or position is to find a new job.

Tip 5: Don’t quit until you have an offer in hand.

Tip 6: Don’t burn bridges. Always aim to leave on good terms with colleagues.

As usual in interviews, they asked about things I’d only vaguely heard of. You could say I knew 50% really well, and the other 50% at a passing level. Luckily, they valued that first 50%.

I worked at Teradata—an American company. But my projects were at the biggest Russian Bank, which meant spending a lot of time in former preschools and tax offices, battling with other consultants and integrators over a decent (non-broken) chair.

Tip 7: Explore all opportunities within the company for personal development.

Even before I started working at Teradata, I knew about the possibility of attending training sessions in Europe. So, by any means necessary, I did my best to attend these trainings. To make it happen, I had to be proactive.

Tip 8: Be proactive. Network with people, seek out opportunities for collaboration, and always look for ways to learn new things.

To be proactive, I started running training sessions for other consultants. I led a few workshops for my colleagues. I also dug through the internal Wiki and found contacts for partner companies. I reached out to everyone, and it led to several meetings and workshops. We even had people fly in from Europe to train us on partner solutions. We didn’t actually use those solutions, and they probably wouldn’t be useful, but I knew they’d add value to my resume and experience.

Through this process, I also discovered that we had a budget for partner courses and exams, which I took full advantage of. Proactivity always pays off.

Job Change – From IT Vendor to a Footwear Startup

According to “Tip 4,” it was time to raise my salary and switch jobs. In this case, the job found me. This was one of those times when proactivity pays off. My former manager asked me to run a training session and later invited me to join their team.

Tip 9: Always have a backup plan. No matter where you’re working, always have a “Plan B.” I like to compare this to climbing a mountain. At any point, we can slip, so every time we make a move forward (upward), we plan several steps ahead. If we make a misstep, we won’t fall because we had a backup plan. Wherever I work, I always look to the future and consider possible scenarios for myself.

It was interesting to work in another international company, but it was the usual story—“no money, but hang in there.” After working there for 1.5 years, I hit the market again and raised my salary to 180,000 rubles. At the time, the exchange rate was 30 rubles to the dollar, which was great. But my excitement didn’t last long—six months later, the ruble crashed, and all my ruble savings for immigration turned from a “carriage” into a “pumpkin.”

Tip 10: Motivate yourself. Often, work can be uninteresting or dull. That’s why you need to create your own reasons for why it’s interesting and why it needs to be done. For example, you’re doing an audit or working in support—boring stuff. Turn this mundane work into a cool story. When interviewing, tell them how you took on tough tasks and what results you achieved.

Job Change – From Moscow to Montenegro

While my documents for Canada were being processed, I interviewed for a Web Analyst position in Montenegro. I had never worked as a web analyst before and had only a surface-level understanding of marketing. Nevertheless, I got the job, and once I arrived, I started learning the ins and outs of digital marketing.

Tip 11: To ace an interview and speak confidently about things you haven’t worked with before, use Steve Jobs’ method (which I learned about fairly recently, though I’ve always used it intuitively): the “reality distortion field.” I previously mentioned “fake it till you make it,” which can sound a bit harsh, like embellishing (at best). But now we have an almost scientific approach from Star Trek—the “reality distortion field.” This means we need to truly believe in what we’re saying.

Tip 12: To start believing in what we’re saying, we need to tell the story at least 15-20 times. So, before tackling serious interviews, practice on “easier targets” by finding simpler companies and roles and telling them your story. Honestly, I’ve long since convinced myself that at the factory, I worked with SQL, databases, and analytics. Wake me up in the middle of the night, and I’ll recount it without a second thought.

I didn’t stay long in Montenegro, as my documents for Canada came through. I had a few interesting interviews in Europe, including one with Accenture. It was a positive experience, but I already had my tickets to Canada.

Tip 13: Interviews with foreign companies are excellent practice for improving your English. So even if you’re not actively seeking a job abroad, why not use the opportunity to practice the language?

Tip 14: Go to interviews anytime and anywhere. Never miss an opportunity to interview, especially if you’re comfortable and not actively looking for a job. It’s better to choose a job without rush or stress, so you can actually choose rather than settle for the first offer that comes along. This way, there’s less need to embellish, and you can be yourself. I saw a LinkedIn post from a guy at Google who asked his mentor about this. His mentor advised the same thing: It’s the employer’s job to provide a comfortable work environment and a competitive salary. If someone else offers you more, they’ll either try to match it to keep you or let you go if they no longer need you. So, everyone benefits. Of course, just make sure no one knows you’re doing it! :)

Job Change – From Montenegro to Canada

Tip 15: If you’re preparing to relocate, start job hunting well in advance—ideally a year or at least 6 months ahead. Subscribe to weekly job listings, connect with people on LinkedIn, network, and introduce yourself. This will save you a lot of time and money down the line.

I arrived in Canada, in Winnipeg, and started my new job at an insurance company within just five days, earning $75,000 CAD, which is about $4,000 a month. I already had an offer in hand, and they were waiting for me.

How did I get the offer remotely? You might guess - I used my own tips;)

How did I find a job even before arriving? The answer is simple: I used my own tips.

Here are some techniques that helped me:

1. I subscribed to job alerts for my province on indeed.ca.

2. I created a resume with a fake local address and set up an IP phone with a Winnipeg area code that forwarded calls to my mobile. My goal was for HR to connect me with the hiring manager, and in the end, my plan worked.

3. Another life hack was to send a Slideshare presentation instead of a cover letter (which I find pretty pointless). Here’s an example, though it’s a bit dated. The main idea was not to showcase my experience and skills but to highlight the value I could add to the company. The key is to focus on the value you bring—identify industry issues, show that you’re aware of them, and outline potential solutions.

Slideshare has a handy feature where you can keep a presentation private, accessible only via a direct link. It also provides web analytics tools, showing you the IP addresses and the number of times your presentation was viewed. This allows you to measure engagement effectively.

Job Change – From Insurance to Amazon

It became clear that Winnipeg wasn’t quite the Canada we had in mind. We needed to move to Vancouver or Toronto. Plus, I had some conflicts with colleagues.

Tip 16: Be cautious with jokes. My harmless jokes weren’t well-received by the team members due to difference of mentalities, who took offense. As a result, I found myself in difficult situations.

I’m grateful to those colleagues for their attempts to get under my skin, as they pushed me to find a job at Amazon. I also had an offer from a local startup, SkipTheDishes, which is quite popular now. I reached out to the founders, went for a meeting, and even brought along my first published book. They offered me a role as Head of Analytics. I took some time to think about it and decided to keep exploring options—which, as it turned out, was the right choice.

Now, I can go into detail about the Amazon interview process. Today, I’m an interview expert, having conducted over 100 interviews at Amazon and completed related trainings. Back then, I knew nothing about the process and was lucky to get through.

Tip 17: If you’re job hunting, use the “carpet bombing” approach—apply to every relevant position and sort through the opportunities later. You might gain valuable negotiation and interview experience, which can help you land your dream job.

I usually keep a spreadsheet where I track all the details:

I had decent experience, a couple of books published, a blog post, but job hunting isn’t a fast process. The main challenge is that each position attracts around 30-50-100 applicants (depending on the company). So, if your resume lands 81st in the list, the chances of them noticing such a valuable candidate are slim. That’s why networking, a quick response time, and a bit of luck are crucial to getting your chance.

I had examples with Google where the recruiter simply vanished because we couldn’t agree on a time. For them, it’s simple—they’re willing to lose a good candidate because, in the end, another strong candidate will come along. Landing a job at a FAANG company is almost like winning the lottery. That’s why finding unconventional ways in is essential.

I got lucky and received an interview invitation.

Tip 18: You need a competitive advantage. What makes you better than other candidates? This can be quickly demonstrated through additional assets—a blog, conference talks, meetups, a YouTube channel. I had books and a blog post, and at Amazon, they valued these artifacts.

Interviewing at Amazon

Amazon Leadership Principles are now part of my DNA.

The interview process typically goes like this:

1. Phone Screen – They call to ask various questions, both technical and non-technical, to see how well you fit.

2. Second Phone Screen – This may be with the hiring manager, and they might also send a test assignment. I was so proactive that I completed the test in three different ways and even created dashboards for them, though they hadn’t asked for it. Afterward, there was a follow-up call about the test.

3. On-Site Interview – If you pass the previous stage, you’ll be invited to an on-site interview (in pre-COVID times). They bought me tickets and scheduled 5-6 interviews, including a lunch with the team.

For the interview, I brought along my signature presentation (my cover letter on Slideshare). The technical director immediately told me it wasn’t necessary and began asking questions, taking notes on my answers. Each interviewer did the same, sometimes asking repeated questions.

Now, I understand what this was about. The interview itself has two main parts:

• Behavioural Interview – Assessing your alignment with Amazon Leadership Principles.

• Functional Interview – Evaluating your technical skills.

The goal of the behavioural interview is to see how well you embody Amazon’s 14 Leadership Principles. Typically, each interviewer focuses on two principles. They’ll select questions from a “question bank,” starting with 2-3 questions to initiate the discussion, then write a report on how well you meet each principle. The aim is to “meet the bar”—meaning you should perform as well as at least 50% of those in this role.

Examples of my questions:

Amazon LP – Ownership

Q: Tell me about a time when you took on something significant outside your area of responsibility. Why was it important? What was the outcome?

Amazon LP – Insist on the Highest Standards

Q: What measures have you personally put in place to ensure performance improvement? What targets and standards were achieved?

The best way to answer these questions is with the STAR format:

• Situation – The context or problem

• Task – The task or challenge

• Action – The actions you took

• Result – The outcome

For the functional interview, they ask technical questions, and there might be a coding session. They asked me a lot about marketing analytics and data warehousing solutions, where my web analytics experience proved useful.

A week later, I got a call with an offer. I didn’t dare negotiate. The salary was low by Amazon’s standards—$90,000 CAD per year, plus a sign-up bonus and 80 Amazon shares over 4 years (worth $600 each at the time, totalling $48,000 CAD over 4 years). I was thrilled, and we moved to British Columbia. They offered me Level 5 (L5).

L5 is considered mid-level. For my first two years, I worked extremely hard, hoping that good performance would lead to a promotion and a salary increase. I was naive.

Tip 19: This one’s obvious, and it’s something we all know but rarely follow: From almost your first day, you need to define clear goals tied to a promotion. These goals may be for the next 12-18 months, but at Amazon, that’s the only way. In my case, I spent two years grinding away, only to be told by my manager, when I asked about a promotion, that I needed to have set these goals in advance.

At Amazon, there’s a performance review once a year, around February. Before this meeting, you need to request feedback from colleagues, who will evaluate you based on two criteria:

1. Your Super Powers (read: Leadership Principles) that you’ve demonstrated.

2. Your Growth Areas (read: Leadership Principles) that you need to improve.

Based on this feedback, your salary may increase by 1-4%. Your manager will explain how they fought hard to make this happen. This increase, pre-tax, might be enough for an extra cappuccino each week.

If you’re lucky, you might also receive additional Amazon stock. Over five years, I received about 25 extra shares beyond the 80 I got on my first day. Of course, this is low—20-40 shares a year is a better benchmark. This largely depends on your manager.

For all five years, whenever I asked, “Why is my salary growing so slowly?” the answer was always that Amazon stock had gone up significantly.

The stock did indeed grow significantly from its initial value. But I fundamentally disagreed with the company’s approach. I received the stock on my first day, and even if I’d only put in minimal effort, I still would have received it. Meanwhile, for my actual performance, overtime, and dedication to the job, I got only a 2-4% raise—essentially just an inflation adjustment.

To get promoted at Amazon, you have to complete several documents, and 60% of the effort must come from your manager. It’s a strategic game that isn’t closely tied to your technical skills and your delivery. That’s why it’s so important (as mentioned in Tip 19) to plan your strategy in advance and discuss it with your manager. The company’s goal is to get the most out of you for the least cost.

Tip 20: In 95% of cases, your salary will remain close to what you initially agreed upon. My salary at Amazon grew by just 10% over five years. Even if I’d been promoted to L6 (the next level), it would only have grown by 10-15%, which is still relatively low. As stated in Tip 4, if you want growth, it’s best to change employers, but make sure you have a solid case and at least 12 months of experience in your current role.

Changing Teams at Amazon

Amazon is a vast corporation with countless teams and business areas. You have the option to transfer to any team worldwide. Many American colleagues moved to Europe, and vice versa—Europe to the U.S. and Canada.

After 2.5 years in one team, I’d had enough of my manager’s empty promises of promotion and decided to look for opportunities in other teams.

Tip 21: This one’s crucial. We often tend to blame others for our failures—thinking, “Everyone else is incompetent; I’m the only smart one.” There are countless examples of this. It’s a terrible mindset to have, and it’s best to avoid this way of thinking. I changed my mindset, learnt to take responsibility for my actions, and stoped considering myself smarter than others.

For example, I fully understand that my inability to get promoted at Amazon was my own failure. So, never think of yourself as better than others. If you know something better, help others understand it. If someone makes a mistake, there’s a reason behind it—help them overcome the issue. But never assume you’re superior; it’ll make life easier and help you achieve your goals.

I was on the West Coast and found a data engineering position on the East Coast.

An interesting paradox: the salary isn’t particularly high, but there are other benefits. For instance, in the summer, I joined the Amazon Alexa team based in Cambridge, MA (turns out there’s a Cambridge in the U.S. too). It’s home to MIT and Harvard, and if you cross the bridge, you’re in Boston.

Our whole family flew to Boston for two months so I could get to know the team. They rented us an apartment just 5 minutes from the office and MIT, costing $12,000 per month. They also covered all our food and my plane ticket. Spending the summer in Boston with family—priceless.

Throughout the year, I made a few more trips to visit them, staying in Airbnb rentals in Boston’s historic center.

Let’s check the next set of Tips!

In addition to Boston, we drove to Seattle every month as a family for Amazon conferences, meetings with colleagues, and other events. As usual, everything was top-notch.

By the way, I first encountered Amazon Alexa when speaking at the Enterprise Data World conference in Boston. It was my second time presenting at this conference; the first was in sunny San Diego. This time, however, it was early March and freezing cold. Speaking at the conference only grants a free ticket. The six-day conference costs $3,500, and speakers attend free but cover their own flights and hotel. I booked a tiny Airbnb room for $100 per night—a 2x2 meter space.

Tip 22: I may have mentioned this before, but I’ll say it again! If you’re serious about your career, say “yes” to every possible opportunity. It was uncomfortable for me for the first few years, but it’s become easier over time. Set yourself a goal—to speak at a meetup, user group, apply for a conference, or share your experience on Surfalytics.com!

Besides conferences, I enjoy doing other things. For instance, I led an Amazon Future Engineer Hour of Code session for my son’s 2nd-grade class, volunteered at a university, and spoke at a high school. I did everything I could. When people asked, “Why?” my answer was that I thought it might come in handy. Every opportunity opens doors to new connections and experiences. If you don’t put yourself out there, nothing will happen.

Tip 23: If you have kids and a family, always look for opportunities to bring them along on trips and conferences. It’s a valuable experience for kids (from ages 5-6 and up) and a great opportunity for your partner.

Communities at Amazon

I’ve already written about proactivity. Almost immediately at Amazon, I began building communities:

• Amazon Tableau User Group – 3,000 users across Amazon who use Tableau. I answered every other question, organized meetings, and invited top-notch speakers.

• BI Tech Talk – 100+ data teams from across Amazon. I also organized presentations with speakers from both Amazon and other companies.

All of this gave me a huge advantage—almost everyone working with data at Amazon knew me. It made switching teams or inviting a speaker from LinkedIn easy. Of course, it didn’t impact my salary, but I enjoyed connecting people and ideas.

Even now, there are people at Microsoft who remember me from when they worked at Amazon.

Tip 24: Always look for collaboration opportunities. Communities are a powerful tool for career growth. Don’t ignore them; create them, help others, and they’ll help you in return.

At Alexa, things went as usual: I did my work but didn’t get closer to a promotion, so I switched teams again. I spoke with teams from California and Seattle and started considering relocating to the U.S. I joined the Customer Behavior Analytics team and promised to relocate to Seattle in the fall of 2020. But I discovered that my Amazon stock grants had run out after four years, and I would have to move my family on a base salary of $130,000, which didn’t meet my needs. The minimum income for Seattle should be $200,000, with a good range being $300-350,000 (including stock), based on my research.

An important point: Amazon prohibits internal transfers from including a salary increase, meant to prevent people from hopping between teams. So each transition sets you back at least a year from a potential promotion.

Regarding bonuses, you’re awarded stock in shares. Multiply the stock price by the number of shares, and you get a nice number. But! When the vested date comes—the date the shares become yours—Canada takes half in taxes. So if you’re granted 20 shares, you only get 10. You can sell them or keep them, as I did. And if you originally had 10 shares at $600 each and they grew to $3,000 each, you’ll owe capital gains tax on $2,400 profit per share, paying 25% of that.

On paper, it looks nice, but in reality, you get much less. Don’t forget about Canada’s progressive income tax—the higher your earnings, the higher the rate. Taxes are lower in the U.S., but they have their own complications with healthcare and insurance.

Every time I was frustrated with my career progress, I’d start job hunting again. Interviewing is beneficial and keeps you sharp. I interviewed with Tesla for a Data Engineer position on their solar team but didn’t get the role. Then I interviewed with Lyft for a Data Engineering Manager role with a California salary of $400-450,000, including bonuses. Although I didn’t have managerial experience, I’d taken all the Amazon management courses. That didn’t work out either. They asked many questions about diversity and inclusion.

For Google, I didn’t even manage to schedule an interview. The recruiter disappeared after the first call. Google can afford to do that—if one good candidate slips away, another will come along.

Tip 25: You’re not the only candidate for the job. Top companies receive hundreds of applications, so try to apply early or seek other routes, like through connections. I applied to dozens of positions at Facebook and Google but received no response.

Eventually, though, recruiters from Facebook reached out to me about a Data Engineering Manager role. Here’s what the interview process looked like:

• Phone Screen with the recruiter – Just a chat about the role, requirements, and my background.

• Leadership Interview – A conversation with a manager, more like a behavioral interview, where they asked about leadership and conflict resolution. Amazon’s management courses had me well-prepared.

• Technical Screening – Even for a manager role, it consisted of three parts: SQL exercises (15 minutes for five tasks), Python (15 minutes for five tasks), and Data Modeling (15 minutes).

To be honest, 15 minutes is very little time. I completed 4/5 SQL tasks and 3/5 Python tasks (which I was proud of, as I hadn’t worked with it much). However, I really struggled with data modeling. The question was simple: imagine you work at LinkedIn; create a model to answer five specific questions. You should start top-down (from business questions to fact tables), but I went bottom-up (from all source data at LinkedIn to fact tables) with only seven minutes left. I also have a theory that North America has a strong lobby of Indian professionals, and I often see teams predominantly made up of them. While their experience and knowledge can vary, my chances felt 50/50. In the end, I didn’t get the job, but it was a great learning experience.

Tip 26: Every failed interview is a huge opportunity for growth. You step out of your comfort zone, prepare, and learn something new. Don’t miss out on this chance. It’s worth it.

Working for “The Man” or for Yourself

Amazon was the pinnacle of my career aspirations. Early on, I couldn’t have dreamed of working there. But after a while, I started to wonder—what’s next? You come to realize that you’re working for “the man,” even if he’s the richest one in the world, Jeff Bezos.

Before the pandemic, I was trying to run two businesses—a consulting company and a meal prep delivery service.

I won’t dive too deeply into my business here. I’ll just say this: if you manage to create a business, service, or product that brings in an income comparable to what you’d earn working for “the man,” you’re on the right track. I didn’t quite succeed. I feel like to build something truly remarkable, you need to fail at 10-15 ventures first. I’ve only failed at two so far. Eventually, burnout set in—that loss of interest in what you’re doing. I was juggling so much at once, and after a few years, I felt I hadn’t achieved anything in return. My career at Amazon hadn’t taken off, my consulting didn’t fly, and my energy was depleted. With the pandemic and working from home, I’d spend entire days at my computer without producing anything.

Another consideration about running your own business is that if you really want to make money, you’ll likely need venture funding, with hopes of an IPO or acquisition. But once you get investors, you’re often back to working for “the man.” Going public is like dreaming your child will make it to the Olympics after joining a sports team.

Then there’s family. I have three kids, and chasing a dream at the expense of family is a high price not everyone is willing to pay. In the end, working for a good company with a decent salary, benefits, and stock options isn’t so bad. But there’s one key aspect—financial planning. A tried-and-true method is investing.

That time we purchased our second home in Greater Vancouver around 2020. In addition, we were able to keep our first home and rent it out.

The approach is as old as time. Of course, to make this work, you need a bit of capital—thanks to Amazon for that. Amazon employees were genuinely able to improve their lives thanks to the company’s growth. Here’s a recent article: Jeff Bezos shared a note from a couple who bought 2 shares of Amazon in 1997—and are now using the proceeds to buy a house after the company’s 172,499% post-IPO growth.

Despite the company being quite tough and demanding, the experience gained there is invaluable.

Tip 27: It may sound simple, but avoid living pay check to pay check. Always think about the future and make an effort to preserve and grow what you have.

Job Change – From Amazon to Microsoft

When I joined my third team at Amazon, I’d promised to relocate to Seattle. But then I realized my stock grants had run out, and no new ones were given. As I mentioned, it’s possible to live on $130,000 in the U.S. with a big family, but it’s not the lifestyle we wanted.

Since I would occasionally apply for jobs on a whim, I applied for a position at Microsoft in Vancouver in May 2020. I didn’t even read the job description (we’d figure it out later, as they say). I applied in May, and they called me at the end of July—2.5 months later. By then, I had almost forgotten about it. The interview process took about a month. From what I gathered, Microsoft doesn’t move too quickly in hiring. The interview process went something like this:

• Phone Interview with HR

• Phone Interview with the manager

These were essentially phone screens. I passed and moved on to the main interview, which is typically an all-day event, but in my case, it was split over two days. Here’s how it went:

1. Interview with the Product Manager and Director – This was a behavioral interview with situational questions. They asked a lot about conflict resolution and collaboration with other teams. I explained everything in detail, trying to identify any Amazon Leadership Principle hints and tailor my answers accordingly.

2. Interview with a Principal and Senior Engineer – They had me solve a statistics problem using SQL and then asked me to design an analytics architecture for Big Data and Streaming on AWS or Azure.

3. Interview with the Data Science and BI team – I shared my ML project experience from my last team, which they found interesting.

After that came the negotiation phase. Based on LinkedIn insights, I found that a Senior Software Engineer in Canada could earn around CAD 220,000 in 2020. But as I’ve seen before, negotiating with Canadian companies can be challenging due to strict limits—they budged only slightly. Either way, I was already “drowning” (burning out) at Amazon, and this was the perfect time for a change. I wasn’t planning to relocate to Seattle either.

In November, I started working at one of the Xbox studios in Vancouver. They set me up with a gaming computer and Xbox, and part of my onboarding process involved playing Xbox games.

When comparing the cultures of Amazon and Microsoft, Microsoft wins on several fronts. The work-life balance, insurance, and benefits are much better.

Comparing AWS and Azure as technologies, I find AWS analytics solutions significantly better suited to my needs.

In terms of salary, Amazon generally pays more. But as you may have gathered, that wasn’t my case. To raise my salary and position, I had to use Tip #4 again—change jobs.

Looking at the market, it’s clear that in Seattle, Amazon and Microsoft employees frequently switch between the two companies. I know people who left Amazon for Microsoft and vice versa, and I’m sure each move came with a raise.

Bonus Round

I almost forgot to mention another interview. Around the same time I was considering Microsoft’s offer, I saw an opening from one of my favorite companies—Slalom (a leading IT consulting firm focused on innovative analytics, design, and development). As I described in my post about my consulting venture, Rock Your Data, Slalom was a source of inspiration for me and served as a benchmark.

They were opening an office in Vancouver and looking for a Director of Data Engineering. It sounded like a dream job, so I applied. I had several interviews with directors from the U.S., and I made a good impression. They weren’t put off by my experience as Data Practice Director at Rock Your Data and my time at Amazon. The only catch was that I knew the technical details a little too well for a director role. A director typically isn’t expected to have such in-depth technical knowledge.

In the end, they told me I didn’t quite fit the director role but offered me a position as a Principal Consultant with a salary 30% higher than Microsoft’s. I consulted my mentor from Amazon, who said not to think twice—go to Microsoft. I was also disappointed by the fact that I’d applied for a director role but received a consultant offer instead.

On the one hand, I was pleased that my consulting experience truly paid off, but on the other, I was disappointed not to land the role I wanted.

Tip 28: Everything happens for the best.

Maybe I was earning a bit less at Microsoft, but I feel good, and I could see my interest in my work coming back.

Tip 29: A high salary isn’t always the main criterion. Consider the long-term prospects, how marketable you’ll be in a year, and whether the role is stable. Will it benefit your mental health and family life? (In my case, consulting and frequent travel clearly weren’t family-friendly.)

This is the ending of my post and tips that I wrote originally in 2021. 100% they are all valid now and it gives your framework to navigate job market.

I would also add couple more tips.

Tip 30: If you have a choice between working for a public company or a startup, set your priorities. If job security and high pay are important, a public company is the better choice, as it offers RSUs, bonuses, and other benefits—they can afford to pay well. If compensation is less important to you and you’d rather focus on developing technical skills, a startup can be a good option. But here’s an important point: don’t join a startup solely because it might go public. The chances of a startup going to IPO are usually very low. Make your decision wisely without counting on the IPO factor.

Continuing to share my experience

I am still learning in the field of data engineering and building Analytics and Data Engineering solutions and still sharing my knowledge and found excellent way via Surfalytics by helping people to leverage 15 years of my experience for the Data Jobs and gain competitive advantage at the Surfalytics.

Tip 31: The more you give to others, the more you receive in return.

In this article, she will cover the foundational knowledge of data visualization, helping you avoid common mistakes and set high standards for data communication with your stakeholders.

Why Data Visualization Matters

Before diving into techniques, let’s start with why visualization is so essential. Data visualization bridges the gap between complex data and comprehensible insight. It helps you see what you can't easily get from the numbers alone. With effective visualizations, you not only make sense of your data but also enable decision-makers to quickly grasp key points.

To highlight this Justin Matejka and George Fitzmaurice created a broader collection of datasets that show how data can produce strikingly different visuals while maintaining similar summary statistics. It was based on the original Anscombe’s Quartet visualizing the same issue.

Each dataset looks unique—whether it's a dinosaur shape or more conventional scatter plots—once graphed. This reinforces that statistics alone aren’t enough to fully understand data.

https://www.autodesk.com/research/publications/same-stats-different-graphs

Both datasets were developed to remind analysts that statistical analysis without visualization can miss critical nuances, patterns, or outliers. Visuals offer the opportunity to catch errors and unlock insights that numbers alone may obscure. That is why always start with visualizing you data.

However it’s always better to try out different charts. I had a very interesting case long time ago where just relying on statistics wasn’t enough, and visualizing the data with a single chart wasn’t sufficient either. We were trying to analyze the results of different blog posts, and one of the metrics was the number of likes. While this use case is fabricated, it’s quite close to reality. As you can see, the average number of likes is higher for Web Design, and the median is higher for Data Science. However, there are two outlier posts in Web Design, which means most posts weren’t performing well, but these two were exceptionally popular. They turned out to be review posts that were indeed very popular. The takeaway is that if we had only relied on averages or medians, we would never have discovered these outliers.