Recent headlines have exposed a critical vulnerability in modern AI: autonomous agents are becoming a prime target for remote execution exploits and data supply-chain attacks. When we define an agent’s runtime behavior using external, unprotected Markdown files, while leaving static credentials exposed in plain text, we inadvertently create a powerful insider threat. The leap from a helpful AI assistant to a rogue execution vector that can corrupt a data warehouse or leak security keys is alarmingly short.

Welcome to the detailed summary and narrative of our latest presentation on AI engineering and security. In this session, we dive deep into the security vulnerabilities of modern autonomous AI agents, demonstrating how easily a helpful assistant can turn into a malicious actor, and laying out the blueprints for securing them using zero-trust architecture.

Below is the complete story, workflow, and code demonstration discussed during the live presentation.

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🤖 AI Agents

Focus: LLM Patterns and Agentic Workflows

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

📉 Machine Learning

Focus: Introduction to machine learning

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

🔗 Related Repository: AI Agents for Data Engineering

To explore the implementation code shown in this presentation, visit the repository: AI Engineering - Agent Development Kit (ADK)

YouTube Video

👍 Subscribe to the channel to get notify on new events!

The Presentation Narrative: The Dark Side of Autonomous Agents

Introduction: The Sealed Illusion

The presentation begins with a reality check about how AI models and agents are deployed in enterprise environments. Platforms like Hugging Face have democratized access to models, acting as repositories similar to how GitHub hosts code. Hugging Face also provides sandboxes where developers can run containers hosting their agent workflows.

However, this deployment model introduces what we call the Sealed Illusion. Developers often believe that because an agent is running inside an isolated Docker container sandbox, restricted by environment variables and standard access tokens, the application is secure.

The flaw in this thinking is that sandboxing only protects against external network intrusion. It does not safeguard against internal prompt manipulation. In modern agent architectures, the instructions governing an agent’s behavior are often loaded dynamically from external markdown files, XML configurations, or block storage. If an attacker manages to modify these instruction files, the container’s perimeter security becomes useless. The attack occurs in-process, shifting the agent’s behavior from the inside out.

The Attack Vector: How Agents Turn Rogue

When an agent’s dynamic specification is altered, its personality changes. An agent designed to act as a helpful Data Analyst can instantly be reprogrammed to act as a Dark Agent. Once the prompt is poisoned, the agent will execute the attacker’s commands using the privileges it has inside the container.

In a typical exploit:

1. The agent reads the poisoned markdown specification.
2. It executes commands to exfiltrate sensitive environment variables, API tokens, and credentials, posting them to a remote hackers’ URL.
3. It crafts and executes destructive actions, such as dropping database tables or wiping out data lakes.

Because the agent is running in-process and has legitimate credentials to access databases or GCS buckets, traditional firewall rules do not stop these queries. The threat is an insider execution exploit triggered by untrusted input.

Architectural Remedy: Policy Enforcement Points (PEP) Architecture

To stop these vulnerabilities, we must establish a zero-trust model using multiple layers of security. We introduce Policy Enforcement Points (PEP) to inspect and validate instructions and tool executions at every stage of the lifecycle.

Layer 1: Context Ingress

Before the agent loads its personality, the incoming prompt specifications must be validated. We implement build-time cryptographic file signatures (.signed.md) using keys retrieved from a cloud vault service. When the agent boots, it checks the cryptographic signature of the specification file. If the file has been modified or the signature is missing, the file is immediately quarantined, and the process halts before the model can ingest it.

Layer 2: Semantic Validation

Once specifications are retrieved, the runtime performs in-process semantic validation. It inspects the structure and contents of the instructions to ensure they have not been hijacked or injected with system overrides before compiling the prompt into active memory.

Layer 3: Pre-Tool Execution Hooks (Allow / Deny / Warn)

Every action an agent performs happens through a tool call (such as a database query or an API call). Before a tool is allowed to execute, we intercept it using pre-tool hooks. These hooks check the requested verbs against security policies:

• ALLOW: The action is safe (e.g., standard SQL selects) and is executed.
• DENY: The action is malicious (e.g., DROP TABLE) and is blocked instantly with a hard permission exception.
• WARN (Human-in-the-Loop): The action is sensitive (e.g., CREATE TABLE or bulk updates) and requires human authorization.

Layer 4: Sidecar Out-of-Process Proxy

As an additional layer of process isolation, we run a sidecar proxy alongside the agent container. If the agent’s internal codebase is compromised, the sidecar proxy intercepts all external communication (e.g., database sockets, GCS queries) and validates the commands against strict network-level policies.

Comparison of Enforcement Layers

Human-in-the-Loop (HITL) Workflow

When a sensitive action triggers a warning policy (Layer 3), we route the request through a secure Human-in-the-Loop approval workflow using an asynchronous semaphore pattern:

1. Halts on Warning: The agent’s execution halts using a semaphore lock.
2. State Persistence: The agent’s current conversation state is persisted to storage.
3. Webhook Notification: An outbound notification (like a Slack webhook) is sent to an administrator with the details of the request.
4. Resumes on Approval: The administrator reviews the action and clicks Approve or Decline, which triggers an inbound API callback to the host process. The host process retrieves the agent state, updates the semaphore, and resumes the agent.

Hands-On Implementation & Remediation (ADK)

Vulnerable Agent vs. Hardened Agent

In a typical Agent Development Kit (ADK) setup, the security posture of the system depends on the validation and hook layers built into the agent class:

• Basic/Tool Agent: Registers standard tools (such as BigQuery or Cloud Storage) but relies on external specifications and environment files without verification.
• Secure Tool Agent: Extends the base agent using object-oriented principles to register cryptographic validation tools, retrieve configuration secrets from cloud vaults, and attach callbacks for pre-tool execution intercept.

Cryptographic Verification, Secret Isolation, and Quarantine

1. Cryptographic Verification (.signed.md): Before parsing any prompt, the agent retrieves the validation signature for the file and checks it against the vault’s cryptographic hashes.
2. Vault Secret Isolation: Secrets are migrated out of local environment files, resolved out-of-scope, and accessed dynamically to prevent active process memory exfiltration.
3. Quarantine States: Files with signature mismatches or missing signatures are quarantined immediately. The agent halts and logs the failure (e.g., fail to load safe asset validation fail. This is a quarantine prompt.), preventing execution.

The Secured Agent Journey: Walkthrough of the Live Code Demonstration

To validate these architectural layers, the ADK provides a local web testing harness that replicates real-world interactions using a web-based chat interface. Although enterprise agents run in headless serverless environments, this harness acts as a crucial debugger for tracing agent execution step-by-step:

1. Establishing the Baseline: The Unprotected Tool Agent

We initialize the system by loading the standard Tool Agent inside the harness. This agent is registered with standard MCP tools, including Google Cloud Storage (GCS) and BigQuery integration.

• The Query: We ask the agent: “Tell us about your governance rules and directives.”
• The Execution: The agent parses a local markdown specification file (tool_agent_instructions.md), maps its operational focus, bucket names, and dataset permissions, and returns a clean, structured overview of its constraints.
• The Vulnerability: The agent assumes the local specification is pristine and trusted. It has no mechanisms to detect if the underlying instructions have been tampered with.

2. Executing the Breach: The Dark Agent Exploit

Next, we load the Dark Agent to simulate a prompt injection exploit.

• The Attack: The Dark Agent is configured with an exploit tool designed to override the system specification. When executed, it silently rewrites tool_agent_instructions.md with a malicious payload.
• The Payload: The modified instructions direct the agent to exfiltrate the container’s environment variables to an external URL and execute destructive DROP TABLE commands in BigQuery.
• The Proof: A Git diff check on the workspace confirms the clean governance rules are replaced with a system override. If the standard Tool Agent runs again, it blindly parses these overrides, shifting its personality to act on behalf of the attacker.

3. Cryptographic Verification & Spec Quarantine

To address this threat, we transition to the Secure Tool Agent, which extends the standard agent’s capabilities with zero-trust validation.

• Signature Check: This agent retrieves the specification file’s signature and compares it with the public keys loaded from a secure vault.
• Quarantine: When initialized with the poisoned configuration, it immediately catches the mismatch. The file signature fails validation, the agent halts, quarantines the compromised document, and outputs: fail to load safe asset validation fail. This is a quarantine prompt.
• Recovery: We restore the instructions from our repository history and perform a system reset. The Secure Agent validates the signature successfully and safely boots.

4. Hardening MCP Tools with Pre-Tool Hooks

Even with file-level security, a compromised memory boundary or prompt bypass could allow malicious commands to reach active tools. We mitigate this using pre-tool-use hooks to intercept all execution requests:

• DENY (SQL Injection Mitigation): When the agent attempts to run a query containing destructive commands (such as DROP TABLE), the hook catches the command in-process, denies execution, and raises a permission exception before the call reaches BigQuery.
• WARN & HITL (Schema Modifications): For sensitive but potentially valid commands (such as CREATE TABLE), the pre-tool hook raises a warning. The runner intercepts this warning, halts execution under a semaphore lock, persists the session context, and alerts administrators via a Slack webhook. Once the administrator approves, the inbound API webhook updates the semaphore state, retrieves the session, and resumes execution safely.

Conclusion: Security Must Be Applied

This presentation highlights a fundamental truth in the era of generative AI: even for autonomous AI agents, security must be applied. Security is not a feature we can delegate entirely to container sandboxes or baseline model boundaries.

The main points of this session underscore that:

• The perimeter is not enough: Dynamic file access and runtime prompt loading create a massive surface area for internal exploits and data exfiltration.
• Verification is essential: Every configuration, specification, and instruction file must be cryptographically signed and verified to establish a secure chain of custody.
• Granular tool policies protect the data plane: Intercepting agent tool calls using pre-tool hooks and Policy Enforcement Points (PEP) allows us to enforce strict ALLOW, DENY, and WARN policies.
• Human-in-the-Loop is a core safety pattern: Persistent state semaphores ensure that sensitive database changes or system modifications remain under human supervision.

Securing AI agents requires the same zero-trust discipline we apply to microservices, APIs, and cloud architecture. By building cryptographic boundaries and semantic safeguards directly into the agent runtime, we can safely unlock the power of agentic automation.

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• News Letter: Sign-up for the newsletter.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

Integrating custom Microsoft Entra ID (Azure AD) authentication with an Azure Static Web App (SWA) on the Standard tier should be straightforward, but subtle misconfigurations can lead to a frustrating infinite redirect loop.

Typically, you log in, complete MFA, and watch the browser bounce repeatedly between Entra ID and your application until Microsoft halts the flow with Error 50074 (“We couldn’t sign you in”). When you check /.auth/me, you find that clientPrincipal is null.

Here is a breakdown of why this happens and the exact checklist to fix it.

The Problem: Why Does the App Loop?

The infinite loop occurs when the interactive user login succeeds in Entra ID, but SWA’s backend proxy fails the subsequent token exchange.

When the token exchange fails:

1. SWA cannot establish the StaticWebAppsAuthCookie.
2. The user lands back on the app unauthenticated (clientPrincipal: null).
3. SWA evaluates the catch-all route guard (/*), issues a 401, and responseOverrides immediately redirects the user back to /.auth/login/aad.
4. The cycle repeats until Entra ID detects the loop and blocks access.

Two common triggers for this failure are explicitly passing OAuth parameters (like response_type=code or scope) inside login.loginParameters—which breaks SWA’s built-in token negotiation—and using internal rewrite rules instead of explicit redirect routes.

What to Check

• SWA Hosting Plan: Ensure your Azure Static Web App is running on the Standard SKU. Custom OpenID Connect / Entra ID identity providers are ignored on the Free tier.
• Platform Type: Configured strictly as Web (not Single-page application) under App registrations > Authentication.
• Redirect URI: Registered under the Web platform as https://<your-domain>/.auth/login/aad/callback.
• ID Tokens Enabled: In Entra ID > Authentication > Platform configurations > Web > Implicit grant and hybrid flows, ensure the checkbox for ID tokens (used for implicit and hybrid flows) is enabled.
• OpenID Issuer Tenant ID: The openIdIssuer URL must use your actual Directory (Tenant) ID (not an Azure Subscription ID), ending with /v2.0 and no trailing slash.
• Client ID Setting: The Application (Client) ID matches the exact environment variable name referenced in clientIdSettingName.
• Client Secret Value: The SWA configuration setting referenced by clientSecretSettingName must hold the Secret Value (the plaintext string generated on creation), not the Secret ID GUID.
• API Permissions: Ensure Microsoft Graph > User.Read (Delegated) is added and granted Admin Consent.
• Enterprise App Assignment: If Assignment required? is set to Yes under Enterprise applications > Properties, confirm the test user or security group is explicitly added under Users and groups.
• No loginParameters Overrides: Omit login.loginParameters entirely from your JSON config. Do not manually pass response_type or scope, as SWA handles Authorization Code negotiation out of the box.
• Route Actions: Set the /login route to an explicit redirect with post_login_redirect_uri=/ rather than an internal rewrite to /.auth/login/aad.
• Clean System Endpoints: Do not define custom route rules in routes for SWA system paths like /.auth/login/aad/callback or /.auth/complete.

Working staticwebapp.config.json Baseline

Use this base configuration to validate that the login process works properly. You can then make additional changes for your configuration.

{
  "auth": {
    "identityProviders": {
      "azureActiveDirectory": {
        "registration": {
          "openIdIssuer": "[https://login.microsoftonline.com/](https://login.microsoftonline.com/)<TENANT_ID>/v2.0",
          "clientIdSettingName": "AZURE_CLIENT_ID",
          "clientSecretSettingName": "AZURE_CLIENT_SECRET"
        }
      }
    }
  },
  "routes": [
    {
      "route": "/login",
      "redirect": "/.auth/login/aad?post_login_redirect_uri=/",
      "statusCode": 302
    },
    {
      "route": "/logout",
      "redirect": "/.auth/logout?post_logout_redirect_uri=/",
      "statusCode": 302
    },
    {
      "route": "/*",
      "allowedRoles": ["authenticated"]
    }
  ],
  "responseOverrides": {
    "401": {
      "redirect": "/.auth/login/aad?post_login_redirect_uri=/",
      "statusCode": 302
    }
  }
}

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

Relying on raw prompt engineering or conversational text interfaces alone will not deliver a production-ready, enterprise-grade AI solution. To build resilient, secure, and compliant software in the age of AI assistants, developers must move past chaotic “vibe coding” and adopt a structured, process-oriented architectural workflow.

This session delivers a blueprint for Specification-Driven Design (SDD), demonstrating how disciplined discovery and modular markdown specifications enable a project assistant to successfully manage and execute multiple phases of the software development lifecycle. By treating AI as an execution engine governed by rigorous technical guardrails, rather than a black-box chatbot, engineers can drastically accelerate velocity while maintaining absolute system integrity. We walk through a real-world case study of a decoupled, zero-trust enterprise quality healthcare risk-assessment engine to show this methodology in action.

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

💡 Contribute: Found a bug or have a suggestion? Open an issue and be part of the open source project.

🔗 Related Repository: AI Engineering

Explore the full implementation of the AI specifications and code used in this workflow: https://github.com/ozkary/ai-engineering

YouTube Video

👍 Subscribe to the channel to get notified on new events!

📅 Agenda

• The Purpose: Understanding the Toolchain & Vibe Coding – Defining the real-world roles of code assistants (like GitHub Copilot and Google Antigravity) and what “vibe coding” actually means for an enterprise workflow.
• The Chaos: The Pitfalls of Planless Execution – The fallout when launching straight into terminal commands without a design baseline: monolithic files, bypassed architectural patterns, and untracked requirements.
• The Discovery & Specification Process: System Decomposition – How to properly break down a complex problem before writing code—identifying distinct system areas, modular boundaries, UI/UX input definitions, core system requirements, and downstream DevOps needs.
• The Blueprint: Establishing Governance & Guardrails – Putting your specifications and governance rules together into a structured, unified layout that forces the AI assistant to strictly follow your target design patterns.
• The Iterative SDLC: Continuous Gap Analysis & Adaptive Specifications – Embracing an Agile approach to handle missing requirements by starting small, continuously feeding refinements to the assistant, and leveraging it to run live gap analyses and solution summaries.

The Purpose: Understanding the Toolchain & Vibe Coding

Modern software development has been dramatically reshaped by AI code assistants like GitHub Copilot and Google Antigravity. As developers, these tools offer immense power, but they require a clear understanding of their role in the developer toolchain. Often, developers fall into the trap of “vibe coding”—sitting down in front of an AI chat interface and typing conversational prompts to generate code on the fly without a predefined plan.

While vibe coding is fast and can yield functional prototypes in a couple of hours, it lacks the structure needed for enterprise-grade applications. For enterprise software, we must treat AI assistants not as autonomous magic boxes that build everything, but as peer programmers. They operate best when provided with clear, contextual instructions, and when their scope is constrained to specific, manageable files or tasks.

By integrating these tools into Visual Studio Code or as command-line interfaces (like the Antigravity CLI), we can work side-by-side with the AI. However, this interaction must be deliberate. We must set guardrails so the AI does not scan unrelated files, dilute its focus, or make arbitrary architectural decisions on our behalf.

Chaos vs. AIStorming

The Chaos: The Pitfalls of Planless Execution

What happens when we dive straight into terminal commands or prompt-driven code generation without a design baseline? We enter a state of planless execution, or chaos.

When you prompt an AI to “build a React application with a product form” without constraints:

1. Monolithic Files: The AI tends to generate a single, massive file containing state, logic, rendering, and styles because that is the most direct path to satisfy the prompt.
2. Bypassed Architecture: Common enterprise architectural patterns, such as separating components, containers, and services, are completely ignored.
3. Untracked Requirements & Lost Context: As the chat history grows, the AI starts losing context. Crucial design constraints, accessibility needs, and security protocols are forgotten, forcing the developer to repeatedly restate them.
4. Untraceability: The final codebase becomes difficult to maintain or trace back to original business requirements. There is no documentation or specifications explaining why decisions were made.

To avoid this chaos, we must establish a rigorous design baseline before writing or generating a single line of code.

AI Storming: The Framework for Structured Discovery

To combat the chaos of planless execution, developers must transition from simple vibe coding to a structured methodology known as AI Storming. Similar to traditional brainstorming sessions where developers sketch out designs and architectural workflows, AI Storming leverages the AI assistant as a cognitive partner to explore the domain model, synthesize requirements, and identify hidden edge cases before a single line of application code is written.

By engaging in this process-oriented framework, you use the AI’s vast knowledge base to analyze project feasibility, perform system decomposition, and quickly get up to speed on unfamiliar technologies or industry standards. Rather than letting the AI write the application directly from conversational chat, you use the AI to draft the specifications that will ultimately govern the build process.

💡 Learn More: To dive deeper into this methodology, read about the AI Storming Framework.

The Discovery & Specification Process: System Decomposition

Enterprise software development begins with discovery and system decomposition. Instead of executing a “big bang” implementation, we break down a complex problem statement into modular, logical boundaries.

For example, when building a real-world healthcare risk-assessment engine:

1. Identify System Areas: Segment the project into distinct areas such as the database, DevOps pipelines, API endpoints, and the user interface (UI/UX).
2. Define Requirements & Compliance: In healthcare, security, accessibility, and zero-PII (Personally Identifiable Information) compliance are paramount. We must explicitly define these constraints up front.
3. Draft Modular Specifications: Create dedicated markdown specifications for each area (e.g., UI specs, API specs, security specs, database schema specs).

By structuring our discovery process and outputting markdown specifications, we create a clear, traceably documented source of truth. This modular approach ensures that both human developers and AI assistants have a clear boundary of work, preventing scope creep and design drift.

The Blueprint: Establishing Governance & Guardrails

Once the modular specifications are drafted, they are unified into a Solution Blueprint. This blueprint serves as the master driver for the AI assistant.

To enforce guardrails and governance:

1. Specify Coding Standards: Define naming conventions (e.g., PascalCase for React components, camelCase for services), CSS libraries (e.g., Tailwind), and state management patterns.
2. Structure the Project Layout: Define the exact directory structure (e.g., /app for React, /api for Python backend, /assets for media) so the AI knows where to locate and create files.
3. Provide Contextual Libraries: If a library is updated or has specific usage guidelines, write those down or pass the official documentation directly to the AI to prevent it from using outdated code patterns.

By feeding these structured specifications to the AI assistant, we establish governance. The AI is forced to work within our design patterns, ensuring that the generated code is consistent, maintainable, and complies with enterprise standards.

The Iterative SDLC: Continuous Gap Analysis & Adaptive Specifications

Specification-Driven Design is not a static, “one-and-done” process. It is an agile, iterative lifecycle.

1. Start with an MVP: Begin with the minimum viable requirements and a baseline specification.
2. Execute and Review: Let the AI generate code based on the specifications. Run the application, review the code structure, and test the features.
3. Conduct Gap Analysis: As implementation progresses, you will inevitably identify missing requirements (e.g., “How do we handle user authentication on Cloud Run?” or “How do we implement voice-to-text hotkeys for accessibility?”).
4. Update the Specs, Not Just the Code: When a gap is found, document the new requirements in the markdown specifications first. Rerun the specs through the AI assistant to let it update the implementation.
5. Traceability & Summaries: Leverage the AI to perform solution summaries and next-step analyses to keep project tracking up-to-date.

This feedback loop ensures absolute traceability. Your documentation and your codebase remain in lockstep, eliminating technical debt and delivering a secure, high-quality, production-ready enterprise solution.

Conclusion

Can we build enterprise-grade, high-quality solutions using generative AI and “vibe coding”? The short answer is yes—but only if we transition from a chaotic conversational interface to a disciplined, process-oriented workflow.

The main highlights of this methodology include:

• Specifications Drive Results: The quality of the AI’s output is directly proportional to the structure of the input. Drafting specifications properly is the most effective way to guide an assistant.
• Process Over Chat: Avoid relying on temporary chat windows where context is easily lost. Maintain permanent, adaptive specs in your repository for absolute traceability.
• AI as a Peer Programmer: Treat AI tools as assistants that execute tasks within clear, modular guardrails rather than a magic wand that does the planning for you.
• Continuous Refinement: Embrace the agile loop of gap analysis, updating specifications, and regenerating code to ensure your code and documentation stay aligned.

Ultimately, by learning how to use AI code assistants properly and wrapping them in rigorous technical governance, developers can achieve massive velocity increases without compromising system integrity or security.

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

As the software development lifecycle shifts toward autonomous vibe coding, rapid AI prototyping, and Software Design Description (SDD) driven workflows, engineering teams face a critical structural bottleneck: the discovery and analysis gap. Generative AI tools excel at emitting syntactically valid code, yet they consistently fail when handed vague, ungrounded, or fragmented domain specifications.

To bridge the gap between initial domain exploration and deterministic engineering execution, I introduce AIStorming—a process-oriented framework born out of my engineering practice and foundational work in enterprise data pipelines and cloud-native application architectures.

Throughout my career designing high-scale cloud-native solutions, event-driven microservices, and modern data platforms, domain research, requirement validation, and exploratory design depended heavily on manual web searches, static documentation reviews, and disconnected whiteboarding sessions. Over years of hands-on practice, my workflow naturally evolved from manual research into AI-driven, interactive, and code-centric discovery. By leveraging AI coding assistants directly inside the development workspace to profile sample datasets, stress-test business rules, probe API interfaces, and validate system constraints in real time, I transformed traditional brainstorming into an active, deterministic engineering phase: AIStorming.

AIStorming formalizes the process of using AI coding tools (e.g., GitHub Copilot, Google Antigravity) to conduct real-time domain discovery, exploratory analysis, and system boundary mapping across both cloud-native software and data engineering domains. Rather than relying on abstract, non-executable IT discovery frameworks, AIStorming anchors domain analysis directly in executable code, version control, and data validation. This yields structured problem statements, refined use cases, quantifiable technical requirements, and machine-readable specifications optimized for vibe coding, cloud-native design, and automated SDD execution.

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

📉 Machine Learning

Focus: Introduction to machine learning

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

1. The Core Engineering Challenge: The Discovery-to-Implementation Gap

In traditional enterprise software engineering, product discovery is often disconnected from actual system implementation. Product managers, business analysts, and architects spend weeks producing static requirements documents, wireframes, or text-heavy stories. When these artifacts are passed down to software engineers—or fed as prompt context into AI coding tools—the implementation breaks down due to implicit assumptions, unverified data schemas, unexpected edge cases, and missing operational constraints.

In an era dominated by AI-assisted synthesis, garbage context in results in garbage execution out.

+-----------------------------------------------------------------------------------+
|                            TRADITIONAL DISCOVERY GAP                              |
|                                                                                   |
|  [ Domain & Business ] ---> ( Static Docs / Whiteboards ) ---> [ AI Assistant ]   |
|  Ideation & Concepts          Unverified Context                 Hallucinated     |
|                                                                  Implementation   |
+-----------------------------------------------------------------------------------+
                                        VS.
+-----------------------------------------------------------------------------------+
|                             AISTORMING FRAMEWORK                                  |
|                                                                                   |
|  [ Domain Input ]  ---> { AIStorming: Code-Centric Analysis } ---> [ SDD & Vibe ] |
|  Raw Data / Rules /     Exploratory Scripts, Profiling,         Deterministic     |
|  API Contracts          & Requirement Synthesis in IDE             Builds         |
+-----------------------------------------------------------------------------------+

To leverage AI coding assistants safely and at enterprise quality, the Discovery Phase must be reinvented. It can no longer be a passive ideation exercise; it must become an active, code-centric discovery engine that validates domain facts, data structures, and service behaviors before full-scale software architecture and implementation begin.

2. What is AIStorming?

AIStorming is the process-oriented methodology of conducting domain discovery, exploratory data/logic analysis, and system boundary definition by pairing human engineering expertise with AI coding tools inside an Integrated Development Environment (IDE).

The objective of AIStorming is not to generate production application code or deploy live systems immediately. Instead, its primary output is a structured set of verified, code-grounded engineering artifacts:

1. Grounded Problem Statements: Unambiguous scope boundaries, operational objectives, and domain invariants.
2. Exploratory Data & Logic Findings: Code-verified insights into payload structures, edge cases, integration contracts, API behaviors, and processing constraints.
3. Formal Functional & Technical Requirements: Explicit inputs, state behaviors, domain events, security boundaries, and non-functional targets (throughput, latency, reliability).
4. Machine-Readable Specifications (SDD): Domain models, schemas, contract interfaces, and prompt-context files tailored for downstream vibe coding, agentic orchestration, and automated pipeline execution.

3. Foundational Principles of AIStorming

Derived from the universal tenets of my Data Engineering Process Fundamentals (DEP) and expanded across cloud-native software architecture, AIStorming adapts proven discovery mechanics to AI-driven developer tooling.

I. The Code-Centric Paradigm

Traditional discovery relies on prose descriptions that fail upon first contact with compiler logic, schema validation, or streaming engines. AIStorming dictates that discovery must be code-centric from day one.

• Exploratory data analysis (EDA), API probing, schema validation, and domain logic simulations are written as executable scripts (Python, TypeScript, SQL, Go, Jupyter Notebooks) inside the IDE workspace.
• AI tools are forced to interact with concrete execution outputs, stack traces, and SDK responses rather than abstract concepts.

II. Dataset & Domain Grounding

AI models hallucinate when operating in an isolated context vacuum. AIStorming requires immediate grounding against real-world sample datasets, schema definitions, domain models, and API interfaces.

• Evaluates data quality, structural variance, integration frequencies (batch vs. streaming event-driven), and service boundaries.
• Uses AI assistants to parse, profile, and transform sample payloads and domain entities live during the discovery window.

III. Source Control and Auditable Iteration

Prompt histories floating in web interfaces are disposable, non-reproducible, and unmaintainable. AIStorming enforces that every discovery artifact—exploratory notebooks, schema models, interface definitions, and context files—is checked directly into a Git repository (e.g., GitHub).

• Enables collaboration between software engineers, data architects, and AI agents.
• Maintains a versioned, auditable history of how business domain rules evolved into formal technical specifications.

IV. Bridge to Vibe Coding and SDD

AIStorming serves as the explicit precursor to Vibe Coding (rapid, intent-driven application development using AI models) and Software Design Description (SDD) generation. By translating loose ideas into code-verified constraints during AIStorming, downstream AI models receive high-fidelity, hallucination-free prompts during full-scale development.

4. The 4 Universal Steps of the AIStorming Process

Whether applied to cloud-native microservices, event-driven streaming applications, serverless architectures, or modern enterprise data platforms, the AIStorming framework executes across four process-oriented phases.

+-------------------+      +-------------------+      +-------------------+      +-------------------+
|  STEP 1:          | ---> |  STEP 2:          | ---> |  STEP 3:          | ---> |  STEP 4:          |
|  Problem          |      |  Exploratory      |      |  Use Case &       |      |  SDD & Vibe       |
|  Framing &        |      |  Data & Domain    |      |  Requirement      |      |  Specification    |
|  Context Prime    |      |  Analysis         |      |  Synthesis        |      |  Output           |
+-------------------+      +-------------------+      +-------------------+      +-------------------+

Step 1: Problem Framing & Context Priming

• Objective: Establish domain boundaries and prime the AI workspace with domain context.
• Process:
  1. Initialize a dedicated discovery branch in the code repository.
  2. Create a core workspace context file (CONTEXT.md or system prompt boundaries) containing raw business objectives, domain rules, SLA requirements, and target application guidelines.
  3. Engage the AI coding assistant to challenge the problem scope, identifying missing assumptions, unstated edge cases, or domain contradictions.

Step 2: Exploratory Data & Domain Logic Analysis

• Objective: Verify domain facts and data behaviors using runnable code inside the IDE.
• Process:
  1. Load representative data samples, schema models, domain event payloads, or third-party API contracts into VS Code or Jupyter Notebooks.
  2. Pair with the AI assistant to write exploratory scripts using standard manipulation libraries, data frames, or contract interfaces (Pandas, Pydantic, Zod, Spark/PySpark, SQL).
  3. Perform data profiling, test payload transformations, inspect edge cases, validate event structures, and evaluate performance/throughput assumptions.
  4. Commit all exploratory scripts, test executions, and output traces to Git.

Step 3: Use Case & Requirement Synthesis

• Objective: Extract structured engineering requirements from discovery findings.
• Process:
  1. Prompt the AI assistant to analyze the commit history, exploratory scripts, and execution outputs generated in Step 2.
  2. Synthesize findings into formalized Use Cases (actor/system interactions, event triggers, preconditions, happy path flows, failover behaviors).
  3. Categorize non-negotiable Technical Requirements:
     • Data Integration & Schema Transformation rules
     • Cloud-Native Application Patterns (Event-driven, REST/GraphQL APIs, Pub/Sub boundaries)
     • Performance, Scalability & Latency SLA targets
     • Security, Identity, Governance, and Compliance boundaries

Step 4: SDD & Vibe Coding Specification Output

• Objective: Produce machine-readable software specifications for autonomous AI execution.
• Process:
  1. Compile discovery outputs into a standardized Software Design Description (SPEC.md / ARCHITECTURE.md).
  2. Generate baseline interface contracts, schemas, and domain types (TypeScript interfaces, OpenAPI specs, Pydantic data models, Avro/Protobuf schemas, ERD models).
  3. Define precise context prompts and agent instructions to drive subsequent implementation phases (Vibe Coding, Automated Test Generation, and CI/CD development).

5. Architectural Deliverables Matrix

To maintain enterprise quality, an AIStorming session must terminate in a concrete set of repository artifacts:

6. Practical Application: Enterprise Cloud-Native & Data Platform Build

To demonstrate the versatility of AIStorming across cloud-native microservices and high-scale data platforms, consider the execution path of a modern enterprise solution:

1. The Human Intent: An architect needs to design a high-throughput, event-driven data ingestion platform that processes live streaming payloads into an analytical data lake.
2. The AIStorming Phase:
   • Instead of asking an AI tool to “write a real-time data ingestion application,” the team opens the IDE and loads representative stream logs and target schemas.
   • The architect uses AIStorming to draft and run exploratory Python/PySpark scripts inside the IDE, testing deserialization speed, schema drift, and payload validation rules.
   • The AI assistant identifies that 6% of incoming event payloads contain missing nested timestamp attributes and schema variations that would break downstream parquet writes.
3. Requirement Synthesis: The team updates REQUIREMENTS.md via AIStorming to explicitly mandate upstream payload sanitization, dead-letter routing (DLQ), and a strict contract enforcement layer.
4. SDD & Vibe Implementation: The resulting SPEC.md, Pydantic models, and OpenAPI/AsyncAPI specs are fed into coding agents. The agents write production-grade microservices and pipeline code on the first pass because the domain edge cases were caught during AIStorming.

7. Conclusion: AIStorming as an Enterprise Engineering Standard

As artificial intelligence shifts software and data engineering from manual syntax writing to high-level system orchestration, the role of the engineer evolves from code writer to system architect and discovery strategist.

AIStorming bridges the foundational gap between raw domain ideas and deterministic execution. By marrying the process-oriented discipline of Data Engineering Process Fundamentals with cloud-native software architecture patterns and modern AI coding tools, AIStorming transforms discovery from a passive, unverified discussion into a repeatable, code-centric, and auditable engineering standard.

By adopting AIStorming as a formal phase prior to vibe coding and SDD execution, engineering organizations can eliminate AI hallucination risks, enforce enterprise domain integrity, and accelerate software delivery with complete architectural control.

References & Foundational Frameworks

• Garcia, Oscar D. (Ozkary). “From Raw Data to Roadmap: The Discovery Phase in Data Engineering Process Fundamentals.” Ozkary Technologies
• Garcia, Oscar D. (Ozkary). “Data Engineering Process Fundamentals - Design and Planning.”

• Garcia, Oscar D. (Ozkary). “Architecting an Agentic Data Pipeline - From Data Lake Discovery to Managed Orchestration.”

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

The goal of this presentation is to introduce the audience to an agentic SDK—specifically the Google Agent Development Kit (ADK), while addressing a critical trap in modern AI engineering. It is incredibly easy to fall into the habit of building AI agents using basic procedural code or siloed Jupyter Notebooks. While these approaches work for initial validation, they fail to scale in an enterprise environment.

Instead, this session demonstrates how to leverage robust software design patterns and foundational architectural principles. By building an abstraction layer over the SDK, we can create a mature, enterprise-ready library. This approach allows us to decouple our core business logic from third-party frameworks, making our codebase completely agnostic to any single SDK and giving us the flexibility to swap underlying tools as the AI ecosystem evolves.

Follow the next sections for the main points of the presentation, and then take a look at the video presentation to dive deeper into the concepts.

Presentation Summary

Discover how to transition from building monolithic, single-prompt chatbots to designing highly modular, scalable, and extendable enterprise agents using the Google Agent Development Kit (ADK). This presentation provides a hands-on architectural deep dive into building process-oriented workflows, implementing the Model Context Protocol (MCP) for cloud data platform integrations, and utilizing automated DevOps tooling to eliminate technical debt in your AI engineering pipelines.

The Monolithic Prompt & SDK Trap

When developers start building AI agents, the initial momentum is almost always driven by quick prototyping. You pull down a hot new SDK, run a quick pip install, and start hardcoding prompts directly into your files.

While this works for a weekend hobby project, it quickly collapses under its own weight in an enterprise ecosystem. You end up with multiple developers writing siloed, inconsistent code, duplicating core tasks like error handling, and introducing massive technical debt. Even worse, your entire system becomes tightly coupled to a single third-party framework. If you ever need to pivot or replace that framework, you are looking at a complete rewrite.

Production-grade engineering requires moving away from spaghetti code toward process-oriented, SDK-agnostic architecture.

Architectural Blueprint: Layered Agent Design

To achieve true reusability and governance, we must build a core architectural foundation that abstracts third-party dependencies away. Instead of letting an external SDK dictate our application structure, we stack agents in specialized layers via inheritance.

1. The Base Agent (The Foundation)

The BaseAgent is an abstract base class responsible for handling cross-cutting concerns that every enterprise agent needs:

• Consistent logging structures.
• Centralized security and exception handling.
• Standardized interface definitions.

By encapsulating these inside a base layer, any new agent you spin up automatically inherits these core enterprise features.

2. The Basic Agent (Configuration over Hardcoding)

The BasicAgent extends the base layer to introduce configuration management. To keep our code robust and maintainable, prompts should never be hardcoded.

Instead, the BasicAgent pulls details—like the target Gemini model or project IDs—from environment variables. Simple instruction hooks can be fed via configurations, allowing your DevOps pipeline to deploy behavior updates or model rollouts without requiring a single line of code to change.

3. The Tool Agent (Advanced Governance & MCP)

The ToolAgent introduces external capabilities through the Model Context Protocol (MCP). For complex enterprise needs, configuration files aren’t enough. The ToolAgent uses file pointers to read advanced Markdown documents detailing strict system instructions, safety limitations, data boundaries, and governance rules.

Extending Capability with Custom & Native MCP Tools

An agent on its own is just an engine that knows how to talk to a Large Language Model. To make it useful, it needs a way to interact with the outside world. This presentation highlighted two separate paradigms for handling tools:

Abstracting Built-in SDK Tools

The Google ADK provides out-of-the-box tools for major platforms like BigQuery. However, to maintain code isolation, we shouldn’t map those tools blindly. In the demo code, we extended the native tool using a custom authorization class (AuthorizationContext). This separation of concerns ensures that token generation, credential refreshing, and cloud authentication happen entirely independent of the agent’s reasoning loop.

Building Custom MCP Tools from Scratch

When an SDK lacks a tool for your specific business requirements—such as interacting with a custom file bucket or specific database—you can build your own using frameworks like FastMCP. The presentation demonstrated a custom Google Cloud Storage (GCS) tool capable of:

• Listing target bucket contents.
• Generating compressed file previews.
• Executing self-diagnostic checks to validate connections before a workflow begins.

The Unsung Hero: The Agent Runner Runtime

One of the least understood components of agentic design is the Agent Runner. While the Google ADK provides an excellent local web-based playground that automatically bootstraps your environment for rapid testing, production environments require you to explicitly script this runtime pipeline.

The Agent Runner acts as the orchestrator of your system, managing three vital elements:

• Orchestration: Connecting multi-agent workflows (e.g., passing tasks seamlessly between a storage agent and a BigQuery data agent).
• Session Management: Directing the active state of an execution path.
• Memory Management: Maintaining persistence. While short-term tasks can run on fast, volatile in-memory sessions, complex industrial or manufacturing pipelines require long-term history to monitor trends, catch system drift, and diagnose process failures over time. The presentation demonstrated wiring an isolated SQLite engine into the runner to handle this tracking cleanly.

Modern DevOps for AI: UV and Makefiles

Enterprise code demands automated quality gates. Rather than relying on standard global package structures, the project repository leverages modern Python tooling to accelerate developer onboarding:

• UV (Virtual Environment Manager): A lightning-fast, modern alternative to legacy virtualenv tools. Using a strict uv.lock file ensures every developer on your team runs identical dependency versions, entirely eradicating the “it works on my machine” problem.

• Makefiles as CI/CD Blueprints: Instead of manually typing tedious execution commands, a standard Makefile orchestrates development tasks. Running make lint catches configuration errors and unmapped dependencies (like an imported but unused session config) before the code ever reaches a code review, while commands like make run-tool smoothly handle localized testing.

Key Takeaways for Enterprise Developers

• Isolate the SDK: Treat third-party agent frameworks as pluggable libraries, not foundational pillars. Abstract them behind abstract base classes so you can swap architectures with minimal friction.
• Configuration Wins Over Code: Keep your agent identities, target models, and governance logic inside Markdown and environment variables. Let your DevOps pipelines drive system behavior.
• Rely on Runtimes for Memory: Keep your agents lean. Let specialized agent runners handle the operational state, session history, and database logging.

Resources & Next Steps

• Get the Code: Explore the foundational structures, base classes, and custom MCP tool definitions by visiting the official GitHub Repository. (Don’t forget to star the repo if you find the patterns helpful!)

• Dive Deeper into Engineering Processes: For a comprehensive guide on building scalable, process-oriented architectural systems, check out my book, “Data Engineering Process Fundamentals”.

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

📉 Machine Learning

Focus: MLOps and Productionizing Models

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

YouTube Video

Take a look at this video and learn about building and testing agents using the Google Agent Development Kit (ADK) by leveraging its CLI and web tool. We’ll start from the absolute basics, learning how to build a simple agent and test it instantly.

From there, we will move on to extend our agents by building custom Model Context Protocol (MCP) tools. Throughout this session, we will focus on staying away from hardcoded prompts. Instead, you’ll learn how to leverage clean software design patterns to build truly reusable, extendable agents that can adapt dynamically with the use of configurable prompts and plug-and-play MCP tools, shifting your development loop from a one-off script or simple notebook into a scalable, production library.

👍 Subscribe to the channel to get notify on new events!

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

Dashboards are effective at showing us that something is breaking, but they usually rely on a human to watch the screen and decide what to do next. In this session, we will look at how to take a standard real-time telemetry dashboard and make it autonomous.

We will walk through a practical implementation using an Angular frontend and a Node.js server. We’ll look at how the system leverages a relational database for persistence and a Redis in-memory cache to handle high-frequency and short volume data feeds. From there, we incorporate an AI Agent that follows three core principles: perceiving the data stream through a sliding window, reasoning against statistical control limits, and acting by sending real-time analysis back to the user. This session is focused on bridging the gap between raw data streams and automated decision-making.

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

📉 Machine Learning

Focus: MLOps and Productionizing Models

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

🔗 Review the repo used for this presentation:

Tech Stack

YouTube Video

👍 Subscribe to the channel to get notify on new events!

📅 Agenda

• The Real-Time Feed: Monitoring Device Telemetry

An introduction to the live system where devices emit high-frequency data for quality monitoring and $3\sigma$ control limit oversight.

• System Architecture: From Ingestion to Persistence

A deep dive into the technical stack, mapping out the data journey through Node.js, Redis in-memory caching, and relational database storage.

• The Human in the Loop: Cognitive Limitations

Exploring the “Passive Monitoring” challenge—why relying on human interpretation of real-time alerts creates a bottleneck in process control.

• The AI Agent: Applying the 3 Principles

Implementing the “Active Observer” using the three core pillars of AI Agents: Perception (the window), Reasoning (the limits), and Action (the analysis).

• The Intelligent Journey: Summary & Advantages

Recapping our transition from a passive monitor to a smart system and discussing the advantages of automated, agentic data interpretation.

⭐ Why Attend?

The industry is moving beyond simple “Chat” interfaces and into the realm of Agentic Observability. By attending this session, you will see a practical blueprint for integrating intelligence directly into a high-velocity data stack.

👥 Who Is This For?

• Junior Developers: Learn the fundamentals of real-time streaming, how to manage state in Node.js, and how to interact with AI APIs in a professional environment.
• Senior Engineers & Architects: See a robust architectural pattern for integrating Redis, relational databases, and AI Agents while maintaining system stability and security.
• Decision Makers (VPs/Directors): Understand the ROI of “Active Monitoring”—how AI Agents can act as a force multiplier for your engineering teams by automating the first layer of data interpretation.
• Quality & Reliability Engineers: Explore how to digitize the control limit logic you already use into an autonomous 24/7 “Digital Twin.”

Presentation

The Real-Time Feed Challenge

High-velocity telemetry defines the “digital pulse” of modern industrial systems.

• Telemetry Streams: Industrial devices emit continuous telemetry (Temperature, Sound, Humidity). These data points require sub-second processing to maintain operational stability.
• Control Limits: Quality engineers rely on statistical boundaries to define “normal” operation. Detecting drift is the critical first step in identifying risks before failure occurs.

Scalable System Architecture

A multi-layered stack designed to bridge the gap between ingestion and intelligence.

• Redis Cache: Manages the “Live State” or Digital Twin. Provides sub-millisecond access for immediate AI perception.
• Relational Data Warehouse: Powers the historical persistence layer for long-term trend analysis and compliance auditing.
• Node Controller: The orchestration hub. Manages telemetry ingestion, state updates, and the execution of the agentic reasoning loop.

The “Human in the Loop” Trap

Passive observability relies on human interpretation, creating a critical bottleneck.

• Cognitive Overload: Humans struggle to interpret hundreds of concurrent streams, missing subtle patterns.
• Alert Fatigue: Constant threshold violations desensitize responders, causing critical $3\sigma$ violations to be ignored.
• Response Latency: The time required for a human to interpret a dashboard often exceeds the window for effective corrective action.

The 3 Principles of AI Agents

Automating observability requires a system that can perceive, reason, and act independently.

1. Perception: Maintaining a stateful sliding window of the Digital Twin to understand temporal context.
2. Reasoning: Evaluating the stream against statistical $3\sigma$ limits and physical engineering constraints.
3. Action: Closing the loop by emitting real-time narratives or triggering autonomous safety protocols.

The Intelligent Journey: Comparison

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

This session explores the strategy of leveraging AI to move beyond manual implementation and into the next level of data engineering. We dive into a process that positions the AI not as a syntax generator, but as a cognitive partner in the engineering lifecycle. We will examine the architectural shift required to transform raw data lake assets into high-performance, orchestrated systems, focusing on the strategic collaboration between human intent and agentic design.

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

📉 Machine Learning

Focus: MLOps and Productionizing Models

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

🔗 Related Repository: AI Agents for Data Engineering

Explore the full implementation of the AI Agents used in this workflow: https://github.com/ozkary/data-engineering-mta-turnstile/tree/main/ai-agents

YouTube Video

👍 Subscribe to the channel to get notify on new events!

📅 Agenda

• Data Lake Discovery: The strategy of deploying discovery agents to autonomously identify patterns and define the foundation of the data grain.
• Governance & Requirements: Establishing the strategic guardrails and requirements that empower an “Architect” agent to maintain system consistency.
• Logical Design for the Staging Area: A process dive into using AI to propose and build a logical abstraction layer, separating raw sources from core business logic.
• Designing and Implementing the Physical Model: How agents navigate the transition to physical storage, building Dimension and Fact tables while maintaining referential integrity.
• Incremental Update Strategy: Developing a sustainable approach to support continuous data feeds from the data lake using idempotent, self-healing processes.
• Pipeline Design and Orchestration: The coordination of complex tasks to manage the relationship between dimensions and facts, ensuring strict lineage and integrated observability.

⭐ Why Attend?

• Elevate Your Role: Learn how to shift your focus from writing repetitive code to defining high-level architectural intent and performing strategic design reviews.
• Master Systemic Reasoning: Understand how to leverage AI to solve complex engineering challenges like referential integrity and dependency management at scale.
• Build for Operations: Move toward a model where system health and observability are built-in byproducts of the design process, not afterthoughts.

👥 Who Is This For?

• Data Engineers & Architects: Looking to evolve their workflow from manual scripting to high-level systemic design.
• Engineering Leaders: Interested in the ROI and reliability of integrating autonomous agents into the development lifecycle.
• AI Enthusiasts: Wanting to see a practical, “beyond-the-chatbot” application of agentic reasoning in a production environment.
• Technical Decision Makers: Seeking a strategy for maintaining governance and referential integrity in an AI-augmented organization.

Presentation

Automating the Data Engineering Lifecycle

We are running a modern Data Engineering process by combining the reasoning power of AI Agents with the standardized connectivity of MCP tools.

• Goal: Move from manual scripting to an intelligent, agent-led pipeline.
• Outcome: A system that can discover, map, and orchestrate data across the cloud.

How do we leverage these tools?

The “Brains” and the “Hands” of the process.

• AI Agents: Use Large Language Models (LLMs) to understand complex system instructions and specific user prompts. They provide the “logic” behind the process.
• MCP Tools: Provide the “connectivity.” They expose metadata to the agent, which allows the AI to understand exactly what actions are available and how to execute them correctly.

How does this all work?

The Execution Loop

• The Model: The agent calls a managed LLM service in the cloud (Gemini) for high-level reasoning.
• Discovery: The agent “sees” the available MCP tools and automatically understands how to use them to interact with GCS or BigQuery.
• Governance: System Prompts provide the guardrails, core requirements, and engineering standards the agent must follow.
• Action: The User Prompt provides the specific task (e.g., “Find today’s files”). The agent then executes the work.

Intelligent Orchestration

We build an AI-powered Data Engineering process that successfully handles:

• Data Lake Discovery: Automatically identifying patterns and namespaces in GCS.
• Data Warehouse Orchestration: Mapping those discoveries directly into BigQuery and creating the data models for analysis.

AI-Driven Data Engineering

• Agents can connect to a data lake and run discovery on the file
• Agents can use the result of the discovery to build external tables, views, tables and even stored procedures for the incremental update process

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

As development teams scale, maintaining architectural consistency becomes the biggest bottleneck. Documents are ignored, and linters only catch syntax errors, not design patterns.

In this session, we will demonstrate how to transform AI from a passive coding assistant into an active Architectural Enforcer. By embedding your “unwritten rules” directly into the repository configuration, you create a developer experience where the AI enforces your patterns in real-time.

We will explore how this shifts the workflow: new developers are guided by the AI from day one, preventing architectural leakage before a pull request is ever opened.

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

📉 Machine Learning

Focus: MLOps and Productionizing Models

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

YouTube Video

👍 Subscribe to the channel to get notify on new events!

Video Agenda

The Problem: Architectural Drift

Why strict rules (Controller-View, Pascal/camelCase) degrade over time and how AI can fix it.

The Intelligence Engine

Breakdown of the core components: Global Rules, Contextual Guardrails, Agent Tools, and Directory Structure.

Configuration: Global Governance

Setting up global “system prompts” for the repository to enforce tech stack and naming conventions.

Configuration: Contextual Guardrails

Creating “firewalls” for specific folders (e.g., preventing logic in views, preventing API calls in Controllers).

Configuration: The Tooling

Building custom Slash Commands (/new-module) to automate “Vertical Slice” scaffolding.

Configuration: The Auditor Agent

Implementing a specialized “Gatekeeper” persona that scans imports to ensure strict layer separation.

Agent Mapping

A conceptual framework comparing repository configuration to autonomous agent architecture.

💡 Why Attend?

• Stop writing boilerplate: Learn to automate complex folder structures with one command.
• Reduce PR Reviews: Shift governance “left” by having the AI catch architectural errors instantly.
• Interactive Demo: See the .github configuration in action on a real codebase.
• Takeaway Code: Leave with the copy-paste markdown templates to implement this in your own repo tomorrow.

Target Audience

• Tech Leads & Architects who need to enforce standards across scaling teams.
• Developers who are tired of correcting the same patterns in code reviews.
• DevOps Engineers interested in “Governance as Code.”
• Leadership teams that are trying to raise standards and productivity in their organizations.

Presentation

SETTING THE STAGE

The Context

• We enforce a strict pattern using the ViCSA architecture
• PascalCase for UI Components.
• camelCase for Logic & Services.
• Separation of Concerns (SoC) is non-negotiable.

The Problem

• Architectural Drift: Patterns degrade over time.
• Passive Docs: Wiki pages are ignored.
• Linter Limits: Linters catch syntax, not architecture.
• Solution: Active Governance via AI.

THE INTELLIGENCE ENGINE

Core AI Policies

• Centralized Config: Rules live in the repo, not the user’s IDE.
• Global Rules: Applied to every interaction (System Prompt).
• Contextual Rules: Triggered only when specific files are opened.
• Agent Tools: Custom commands to scaffold new components, controllers or services.

CONFIGURATION: GLOBAL GOVERNANCE

Global Instructions

File: .github/copilot-instructions.md

This acts as the System Prompt for the entire repository. It is silently added to every interaction.

• Tech Stack: TS, Tailwind, Hooks.
• Naming: Pascal vs camelCase.
• Flow: View → Controller → Service -> API.

DEV EXPERIENCE: THE SILENT ENFORCER

Without Config

A developer asks:

How do I create a new service?

• AI suggests a generic Class-based service.
• Suggests creating a utils.js file.
• Ignores project folder structure.

With Config

A developer asks: How do I create a new service?"

• AI reads the Governance.
• Response: Create src/services/userAuth/index.ts using a functional export, as per project standards.

CONFIGURATION: CONTEXTUAL GUARDRAILS

View Layer Rules

File: .github/instructions/controller-layer.md

Trigger: Opening any **/*.tsx file.

• “You are a View.”
• “No Logic allowed.”
• “No direct API calls.”

Controller Layer Rules

File: .github/instructions/view-layer.md

Trigger: Opening any **/controller.ts file.

• “You are a Controller.”
• “Use Services, NOT Fetch.”
• “Manage State here.”

DEV EXPERIENCE: REAL-TIME INTERVENTION

The Scenario

• A developer tries to write fetch() inside a UI Component (index.tsx).
• They ask Copilot: “Write a fetch call here for me.”

The Intervention

Ghost Text: Copilot refuses to autocomplete the network call.

Chat Reply:

I cannot. This is a View file. Please move this logic to the sibling Controller (index.ts) and import it.

CONFIGURATION: THE TOOLING

Prompt Library

File: .github/prompts/new-module.md

These act as Agent Tools or “Slash Commands”.

• Goal: Automate the “Vertical Slice”.
• Benefit: Complex scaffolding logic is stored in the repo, not in the developer’s head.
• Usage: /new-module

# Prompt Library (The Scaffolder)
File: `.github/prompts/new-component.md`
Goal: Automate the creation of a standalone UI Component with optional Service/API layers.

# Create New Component
I need to generate a new component following our **Folder-as-Namespace** pattern.
**Command:** `/new-component: `

Please generate the code blocks for the layers requested in the arguments (service, api). 
*Note: Logic folders must be camelCase. UI folders must be PascalCase.*

---

### Component Layer (Required)
**Folder:** `src/components//`
- **File:** `controller.ts` (Controller): Logic and State only.
- **File:** `index.tsx` (View): Pure UI. Imports Controller.
---


### Service Layer (Optional)
*Condition: Generate only if 'service' is present in .*

**File:** `src/services//index.ts`
- **Role:** Business logic and data transformation.
- **Code:** Import the API (if requested). Export a service object or functional exports.

---

### API Layer (Optional)
*Condition: Generate only if 'api' is present in .*

**File:** `src/apis//index.ts`
- **Role:** Define specific endpoints.
- **Code:** Import `coreClient` from `src/apis/index.ts`. Export async functions with typed responses.

---

### Style Guidelines
- **Typing:** Use TypeScript interfaces for all Props and Data models.
- **Separation:** Logic stays in `controller.ts`, JSX stays in `index.tsx`.
- **Naming:** Components use PascalCase; Services/APIs use camelCase.

DEV EXPERIENCE: THE SCAFFOLDING

The Command

Starting a new feature called “Sales Dashboard”.

Action:

/new-module featureName:Sales Dashboard

The Execution

• Analyzes the request.
• Applies PascalCase to Containers/Components folders.
• Applies camelCase to api/service folders.
• Generates the Controller-View pair instantly.

THE RESULT: GENERATED ARCHITECTURE

The Results

• Layers generated instantly.
• Correct naming conventions applied.
• Zero manual boilerplate.

CONFIGURATION: THE AUDITOR AGENT

Specialized Persona

File: .github/agents/arch-auditor.md

This creates a named Agent that acts as a Gatekeeper. It doesn’t write features; it verifies them.

• Role: Architecture Enforcer.
• Task: Scans imports to ensure strict layer separation.
• Rule: “Views never talk to APIs.”

# Custom AI Agent (The Reviewer)
Agent ID: `@vicsa-auditor`

Context: A bot that ensures the chain of command is respected using the ViCSA architecture (View Controller Service API)

## Primary Objective
name: Architecture Auditor
description: Verifies strict separation of Controller, Service, and View layers.
tools: [code-search]

---
## Role
You ensure the integrity of the data flow: View -> Controller -> Service -> API.

## Audit Logic
When asked to "Audit this feature":

1. **Check the View (.tsx):** - FAIL if it imports `src/services`.
   - FAIL if it imports `src/apis`.
   - PASS only if it imports `./index`.

2. **Check the Controller (.ts):**
   - FAIL if it uses `fetch` or `axios`.
   - PASS only if it delegates to `src/services`.

3. **Check the Service:**
   - FAIL if it defines its own URL logic.
   - PASS only if it imports `src/apis/index.ts`.

DEV EXPERIENCE: THE CODE REVIEW

The Interaction

Before raising a pull request, the developer invokes the auditor.

Prompt:

@vicsa-auditor check this component for violations.

Response:

✅ PASS: SalesDashboard/index.tsx imports only from its sibling controller. No direct API calls found.

THE AUTONOMY ADVANTAGE

AI enforces the ViCSA architecture through continuous observation and autonomous execution.

• Perception: Continuously observes the active workspace, file paths (e.g., src/components/), and context to understand the developer’s structural intent.
• Reasoning: Evaluates the perceived context against the repository’s .github Guardrails, determining if a View is bypassing a Controller or violating Separation of Concerns, SoC.
• Action: Executes autonomous scaffolding, enforces strict ViCSA governance, provides recommended fixes feedback.

SUMMARY & AGENT MAPPING

Embedding governance directly into the repository transforms the development lifecycle. It replaces passive wiki pages with active, real-time enforcement, ensuring that every AI suggestion aligns with architectural standards. This eliminates “drift”, accelerates onboarding, and turns Copilot into a domain-expert partner.

RAG: Retrieval augmented generation

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

In the modern data landscape, the wall between “where data lives” and “how we get insights” is crumbling. This session focuses on the Cognitive Data Lakehouse. A paradigm shift that allows developers to treat a fragmented data lake as a unified, high-performance warehouse.

We will explore how to move beyond brittle ETL pipelines using Zero-ETL architecture in the cloud. The core of our discussion will center on using integrated AI capabilities and semantic modeling to solve the “Metadata Mess” inherent in global manufacturing feeds without moving a single byte of data. From raw telemetry in object storage to semantic intelligence via large language models, we’ll show you the real-world application of AI in modern data engineering.

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

📉 Machine Learning

Focus: MLOps and Productionizing Models

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

YouTube Video

Video Agenda

Phase 1: Foundations & The Zero-ETL Strategy

We kick off with the infrastructure layer. We’ll discuss the design of cross-region telemetry tables and how modern cloud engines allow us to query raw files in object storage with the performance of a native table. We’ll establish why “0x data movement” is the goal for modern scalability.

Phase 2: Confronting the Metadata Mess

Schema drift and inconsistent naming across global regions are the enemies of unified analytics. We will look at why traditional manual mapping fails and how we can use AI inference to bridge these gaps and standardize naming conventions automatically.

Phase 3: AI-Driven Unification & Semantic Modeling

The “Cognitive” part of the Lakehouse. We’ll dive into the technical implementation of registering AI models directly within your data warehouse environment. You’ll see how to create an abstraction layer that uses AI to normalize data on the fly, creating a robust semantic model.

Phase 4: Scaling to a Global Feed

Finally, we’ll demonstrate the DevOps workflow for integrating a new international factory feed into a global telemetry view. We’ll show how to maintain a “Single Source of Intelligence” that BI tools and analysts can consume without needing to know the complexities of the underlying lake.

💡 Why Attend?

• Master Modern Architecture: Learn the “Abstraction Layer” design pattern that is replacing traditional, slow ETL/ELT processes.
• Hands-on AI for Data Ops: See exactly how to use AI and semantic modeling within SQL-based workflows to automate data cleaning and schema mapping.
• Scale Without Pain: Discover how to manage global data sources (multi-region, multi-format) through a single governing layer.
• Developer Networking: Connect with other data architects, engineering leaders, and professionals solving similar scale and complexity challenges.

Target Audience: Data Engineers, Analytics Architects, Cloud Developers, and anyone interested in the intersection of Big Data and Generative AI.

Presentation

Phase 1: The Zero-ETL Strategy

INFRASTRUCTURE: DATA STAYS LOCAL

Architecting for Scale

• Storage Decoupling: Raw files remain in the Data Lake, eliminating replication overhead.
• Virtual Access: Data Warehouse external tables allow immediate querying of CSV, Parquet, and JSON.
• Minimal Latency: No waiting for ingest pipelines; analysis starts upon file arrival.

UNMATCHED STORAGE EFFICIENCY

Zero Data Replication

• Traditional ETL requires moving data across multiple tiers. Our architecture ensures a single source of truth with zero data movement between GCS and BigQuery compute.
• This is similar to the Bronze Zone in a Medallion Architecture.

Phase 2: The Metadata Mess

CHALLENGES OF UNIFICATION

Schema Friction

• Feeds arrive with inconsistent headers (e.g., ‘Device Number’ vs ‘deviceNo’). Manual aliasing is fragile and slow.

Entity Drift

• Names and IDs vary across systems, preventing standard joins from matching records effectively.

Type Mismatches

• Varying data types for the same concept (Integer vs String) crash standard SQL aggregation views.

Phase 3: The AI Solution

BIGQUERY STUDIO: THE AI INTERFACE

Remote AI Registration

• Register Gemini Pro directly inside BigQuery to enable cognitive functions within your SQL workspace.

CREATE MODEL `gemini_remote`
REMOTE WITH CONNECTION `bq_connection`
OPTIONS(endpoint = 'gemini-1.5-pro');

Automated Inference

• AI “reads” information schemas to infer mapping logic, moving you from Code Author to Logic Approver.

SELECT ml_generate_text_result
FROM ML.GENERATE_TEXT(
  MODEL `gemini_remote`,
  (SELECT "Compare Source A and B schemas. Write a SQL view to unify them." AS prompt)
);

AI-ASSISTED SCHEMA DISCOVERY

Prompting for Base Tables

• Using AI to generate the DDL for external tables by pointing to compressed feeds in the lake (USA & MEX factories).

SELECT ml_generate_text_result
FROM ML.GENERATE_TEXT(
  MODEL `gemini_remote`,
  (SELECT "Create External Tables as smart_factory.us_telemetry with path 'gs://factory-dl/us/dev-540/telemetry-*.csv.gz' '. Include option CSV, GZIP compression and skip 1 row. Infer and add the schema using lower case" AS prompt));

SELECT ml_generate_text_result
FROM ML.GENERATE_TEXT(
  MODEL `gemini_remote`,
  (SELECT "Create External Tables as smart_factory.mx_telemetry with path 'gs://factory-dl/mx/dev-940/telemetry-*.csv.gz' '. Include option CSV, GZIP compression and skip 1 row. Use schema device_number STRING, bay_id INT64, factory STRING, created STRING" AS prompt));

Generated BigLake DDL

-- USA Factory Feed
CREATE OR REPLACE EXTERNAL TABLE `smart_factory.us_telemetry` (
  device_number STRING,
  bay_id INT64,
  factory STRING,
  created STRING
)
OPTIONS (
  format = 'CSV',
  uris = ['gs://factory-dl/us/dev-540/telemetry*.csv.gz'],
  skip_leading_rows = 1,
  compression = 'GZIP'
);

-- MEX Factory Feed
CREATE OR REPLACE EXTERNAL TABLE `smart_factory.mx_telemetry` (
  device_number STRING,
  bay_id INT64,
  factory STRING,
  created STRING
)
OPTIONS (
  format = 'CSV',
  uris = ['gs://factory-dl/mx/dev-940/telemetry*.csv.gz'],
  skip_leading_rows = 1,
  compression = 'GZIP'
);

AI-ABSTRACTION: THE VIEW LAYER

Generating the Interface

• AI creates a clean abstraction view for each external table, decoupling raw storage from the analytics model.

-- AI Instruction
"Create a view named 
smart_factory.vw_us_telemetry 
selecting all columns from the
usa_telemetry table. Safe cast the created column as datetime."

Abstraction Layer DDL

-- Semantic Abstraction Layer
CREATE OR REPLACE VIEW `smart_factory.vw_us_telemetry` AS
SELECT 
  device_number,
  bay_id,
  factory,
  SAFE_CAST(created as DATETIME) AS created
FROM `smart_factory.us_telemetry`;

COGNITIVE UNIFICATION

The Multi-Region Model

• The unified view now consumes from the abstraction layer, ensuring that changes to raw storage don’t break the views down stream.

-- AI Instruction
"Create a view with name
smart_factory.vw_telemetry that creates a union of all the fields from the views vw_[region]_telemetry. The regions include us and mx. List out all the field names. Never use * for field names"

Unified Global View

-- Semantic Abstraction Layer
CREATE OR REPLACE VIEW `smart_factory.vw_telemetry` AS
SELECT 
  device_number,
  bay_id,
  factory,
  created
FROM `smart_factory.vw_us_telemetry`
UNION ALL
SELECT 
  device_number,
  bay_id,
  factory,
  created
FROM `smart_factory.vw_mx_telemetry`

SCALING TO CHINA FACTORY

Evolving the Model

• Adding the new China feed by generating the External Table definition via AI.

CREATE OR REPLACE EXTERNAL TABLE `smart_factory.cn_telemetry` (
  device_number STRING,
  bay_id INT64,
  factory STRING,
  created STRING
)
OPTIONS (
  format = 'CSV',
  uris = ['gs://factory-dl/cn/dev-900/telemetry*.csv.gz'],
  skip_leading_rows = 1,
  compression = 'GZIP'

Human-in-the-Loop DevOps

• Use AI to update the unified view with the new data feed. Review and apply the changes by the DevOps team, as changes to a production view require approval.

Manufacturing SPC & Root Cause Analysis

• This query calculates a rolling mean and standard deviation over the last 10 minutes of telemetry to detect anomalies, “Out of Control” conditions.

WITH TelemetryStats AS (
  SELECT
    machine_id,
    timestamp,
    sensor_reading,
    -- Calculate rolling stats for the "Control Chart"
    AVG(sensor_reading) OVER(PARTITION BY machine_id ORDER BY timestamp ROWS BETWEEN 20 PRECEDING AND CURRENT ROW) as rolling_avg,
    STDDEV(sensor_reading) OVER(PARTITION BY machine_id ORDER BY timestamp ROWS BETWEEN 20 PRECEDING AND CURRENT ROW) as rolling_stddev
  FROM `production_data.mx_telemetry_stream`
  WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
),
Anomalies AS (
  SELECT *,
    -- Define "Out of Control" (Reading > 3 Sigma from mean)
    ABS(sensor_reading - rolling_avg) > (3 * rolling_stddev) AS is_out_of_control
  FROM TelemetryStats
)
SELECT * FROM Anomalies WHERE is_out_of_control = TRUE;

Control Chart Visualization

ADVANTAGE COMPARISON MATRIX

Strategic Intelligence ROI:

ROI(ai) = Insights Velocity / (Movement Cost + Labor Hours)

FINAL THOUGHTS: STRATEGIC SUMMARY

Legacy Challenges

• Brittle ETL: Manual pipelines break with every schema change.
• Cost Inefficiency: Redundant storage for processed data.
• Semantic Silos: Hard-coded aliases for disparate naming conventions.
• Slow Time-to-Insight: Weeks spent on manual schema alignment.

AI-Assisted Solutions

• Zero-ETL Arch: Cost-effective storage with Data Lake virtual access.
• Automated Inference: Vertex AI handles the “heavy lifting” of mapping.
• Adaptive DevOps: Scalable model evolution (USA → MEX → China).
• Unified Intelligence: One virtual source of truth for global analytics.

Moving from Data Reporting to Active Semantic Intelligence.

We’ve covered a lot today, but this is just the beginning!

If you’re interested in learning more about building cloud data pipelines, I encourage you to check out my book, ‘Data Engineering Process Fundamentals,’ part of the Data Engineering Process Fundamentals series. It provides in-depth explanations, code samples, and practical exercises to help in your learning.

📅 Upcoming Sessions

Our upcoming series expands beyond data engineering to bridge the gap between AI, Machine Learning, and modern cloud architecture. Using our Data, AI, and ML GitHub blueprints, we provide the code-first patterns needed to build everything from Zero-ETL pipelines to scalable LLM-powered systems. Join us to explore how these integrated disciplines work together to turn raw data into production-ready intelligence.

🌟 Let’s Connect & Build Together

If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

Build upon your existing data engineering expertise and discover how Medallion Architecture can transform your data strategy. This session provides a hands-on approach to implementing Medallion principles, empowering you to create a robust, scalable, and governed data platform.

We’ll explore how to align data engineering processes with Medallion Architecture, identifying opportunities for optimization and improvement. By understanding the core principles and practical implementation steps, you’ll learn how to optimize data pipelines, enhance data quality, and unlock valuable insights through a structured, layered approach to drive business success.

• Follow this GitHub repo during the presentation: (Give it a star)

👉 https://github.com/ozkary/data-engineering-mta-turnstile

• Read more information on my blog at:

👉 https://www.ozkary.com/2023/03/data-engineering-process-fundamentals.html

YouTube Video

Video Agenda

• Introduction to Medallion Architecture
  • Defining Medallion Architecture
  • Core Principles
  • Benefits of Medallion Architecture
• The Raw Zone
  • Understanding the purpose of the Raw Zone
  • Best practices for data ingestion and storage
• The Bronze Zone
  • Data transformation and cleansing
  • Creating a foundation for analysis
• The Silver Zone
  • Data optimization and summarization
  • Preparing data for consumption
• The Gold Zone
  • Curated data for insights and action
  • Enabling self-service analytics
• Empowering Insights
  • Data-driven decision-making
  • Accelerated Insights
• Data Governance
  • Importance of data governance in Medallion Architecture
  • Implementing data ownership and stewardship
  • Ensuring data quality and security

Why Attend:

Gain a deep understanding of Medallion Architecture and its application in modern data engineering. Learn how to optimize data pipelines, improve data quality, and unlock valuable insights. Discover practical steps to implement Medallion principles in your organization and drive data-driven decision-making.

Presentation

Introducing Medallion Architecture

Medallion architecture is a data management approach that organizes data into distinct layers based on its quality and processing level.

• Improved Data Quality: By separating data into different zones, you can focus on data quality at each stage.
• Enhanced Data Governance: Clear data ownership and lineage improve data trustworthiness.
• Accelerated Insights: Optimized data in the Silver and Gold zones enables faster query performance.
• Scalability: The layered approach can accommodate growing data volumes and complexity.
• Cost Efficiency: Optimized data storage and processing can reduce costs.

The Raw Zone: Foundation of Your Data Lake

The Raw Zone is the initial landing place for raw, unprocessed data. It serves as a historical archive of your data sources.

• Key Characteristics:
  • Unstructured or semi-structured format (e.g., CSV, JSON, Parquet)
  • Data is ingested as-is, without any cleaning or transformation
  • High volume and velocity
  • Data retention policies are crucial
• Benefits:
  • Preserves original data for potential future analysis
  • Enables data reprocessing
  • Supports data lineage and auditability

The Bronze Zone: Transforming Raw Data

The Bronze Zone is where raw data undergoes initial cleaning, structuring, and transformation. It serves as a staging area for data before moving to the Silver Zone.

• Key Characteristics:
  • Data is cleansed and standardized
  • Basic transformations are applied (e.g., data type conversions, null handling)
  • Data is structured into tables or views
  • Data quality checks are implemented
  • Data retention policies may be shorter than the Raw Zone
• Benefits:
  • Improves data quality and consistency
  • Provides a foundation for further analysis
  • Enables data exploration and discovery

The Silver Zone: A Foundation for Insights

The Silver Zone houses data that has been further refined, aggregated, and optimized for specific use cases. It serves as a bridge between the raw data and the final curated datasets.

• Key Characteristics:
  • Data is cleansed, standardized, and enriched
  • Data is structured for analytical purposes (e.g., normalized, de-normalized)
  • Data is optimized for query performance (e.g., partitioning, indexing)
  • Data is aggregated and summarized for specific use cases
• Benefits:
  • Improved query performance
  • Supports self-service analytics
  • Enables advanced analytics and machine learning
  • Reduces query costs

The Gold Zone: Your Data’s Final Destination

• Definition: The Gold Zone contains the final, curated datasets ready for consumption by business users and applications. It is the pinnacle of data transformation and optimization.
• Key Characteristics:
  • Data is highly refined, aggregated, and optimized for specific use cases
  • Data is often materialized for performance
  • Data is subject to rigorous quality checks and validation
  • Data is secured and governed
• Benefits:
  • Enables rapid insights and decision-making
  • Supports self-service analytics and reporting
  • Provides a foundation for advanced analytics and machine learning
  • Reduces query latency

The Gold Zone: Empowering Insights and Actions

The Gold Zone is the final destination for data, providing a foundation for insights, analysis, and action. It houses curated, optimized datasets ready for consumption.

• Key Characteristics:
  • Data is accessible and easily consumable
  • Supports various analytical tools and platforms (BI, ML, data science)
  • Enables self-service analytics
  • Drives business decisions and actions
• Examples of Consumption Tools:
  • Business Intelligence (BI) tools (Looker, Tableau, Power BI)
  • Data science platforms (Python, R, SQL)
  • Machine learning platforms (TensorFlow, PyTorch)
  • Advanced analytics tools

Data Governance: The Cornerstone of Data Management

Data governance is the framework that defines how data is managed within an organization, while data management is the operational execution of those policies. Data Governance is essential for ensuring data quality, consistency, and security.

Key components of data governance include:

• Data Lineage: Tracking data’s journey from source to consumption.
• Data Ownership: Defining who is responsible for data accuracy and usage.
• Data Stewardship: Managing data on a day-to-day basis, ensuring quality and compliance.
• Data Security: Protecting data from unauthorized access, use, disclosure, disruption, modification, or destruction.
• Compliance: Adhering to industry regulations (e.g., GDPR, CCPA, HIPAA) and internal policies.

By establishing clear roles, responsibilities, and data lineage, organizations can build trust in their data, improve decision-making, and mitigate risks.

Data Transformation and Incremental Strategy

The data transformation phase is a critical stage in a data warehouse project. This phase involves several key steps, including data extraction, cleaning, loading, data type casting, use of naming conventions, and implementing incremental loads to continuously insert the new information since the last update via batch processes.

Data Lineage: Tracks the flow of data from its origin to its destination, including all the intermediate processes and transformations that it undergoes.

Data Governance : Metadata

Assigns the owner, steward and responsibilities of the data.

Summary: Leverage Medallion Architecture for Success

• Key Benefits:
  • Improved data quality
  • Enhanced governance
  • Accelerated insights
  • Scalability
  • Cost Efficiency.

We’ve covered a lot today, but this is just the beginning!

If you’re interested in learning more about building cloud data pipelines, I encourage you to check out my book, ‘Data Engineering Process Fundamentals,’ part of the Data Engineering Process Fundamentals series. It provides in-depth explanations, code samples, and practical exercises to help in your learning.

Upcoming Talks:

Join us for subsequent sessions in our Data Engineering Process Fundamentals series, where we will delve deeper into specific facets of data engineering, exploring topics such as data modeling, pipelines, and best practices in data governance.

This presentation is based on the book, Data Engineering Process Fundamentals, which provides a more comprehensive guide to the topics we’ll cover. You can find all the sample code and datasets used in this presentation on our popular GitHub repository Introduction to Data Engineering Process Fundamentals.

Thanks for reading! 😊 If you enjoyed this post and would like to stay updated with our latest content, don’t forget to follow us. Join our community and be the first to know about new articles, exclusive insights, and more!

• Google Developer Group
• GitHub
• Twitter
• YouTube
• BlueSky

👍 Originally published by ozkary.com