Your agent's persona was never meant to carry your entire domain knowledge. As teams bring AI into real data engineering workflows—handling shifting ingestion feeds, inferring schemas, and standardizing lakehouse partitions—a single system instructions file quickly becomes a tangled mess of identity, procedure, and governance. It becomes difficult to maintain, costly to run, and impossible to trust in production pipelines.
The solution lies in classic software architecture principles: skills teach an agent what it knows, while commands give it something it can do. By decoupling domain judgment from deterministic execution, we build modular, testable, and cost-effective AI systems. A deterministic data pipeline command executes predictable steps automatically, invoking an AI skill only when genuine reasoning is required—such as when an unfamiliar payload arrives with an unmapped schema that threatens downstream tables.
In this post, we explore the architectural patterns behind modular agents, moving from an overburdened monolithic persona to a decoupled system that integrates human-in-the-loop security gates, Model Context Protocol (MCP) tools, and automated schema drift resolution.
Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!
Focus: LLM Patterns, Skills & Commands, and Agentic Workflows
Focus: Real-world ETL & MTA Turnstile Data
Focus: Architectural patterns, hands-on data pipelines, and MTA turnstile case studies
💡 Contribute: Found a bug or have a suggestion? Open an issue and be part of the open source project.
Explore the full implementation of the skills, commands, and agent definitions used in this workflow:
https://github.com/ozkary/ai-engineering/tree/main/adk
👍 Subscribe to the channel to get notified on new events!
When engineers first build AI agents, the standard approach is straightforward: open a prompt file or system message configuration and draft a comprehensive persona. This single file frequently includes:
While this works for simple prototypes, in an enterprise data pipeline it becomes a severe anti-pattern known as the Monolithic Agent.
graph TD
subgraph Monolithic Agent Anti-Pattern
A[Single System Prompt] --> B[Domain Knowledge: MTA / Telemetry]
A --> C[Dialect Rules: BigQuery / Snowflake]
A --> D[Execution Logic: Table DDL / Storage]
A --> E[Governance & Security Constraints]
end
B & C & D & E --> F[Context Bloat & Token Burn]
F --> G[Hallucinations & Pipeline Crashes]
To solve this, we decouple the agent into Skills (declarative knowledge) and Commands (deterministic actions).
A Skill represents domain expertise and semantic judgment. It answers the question: What does the agent know?
In practice, a skill is a structured specification (typically a modular Markdown manifest) that defines the rules, schema specifications, and analytical judgment required for a specific business domain.
graph TD
subgraph Skills Architecture
A[Agent Kernel]
A -->|Injected via DI| B[MTA Transit Skill]
A -->|Injected via DI| C[Factory Telemetry Skill]
A -->|Injected via DI| D[BigQuery Governance Skill]
B & C & D --> E[Domain Judgment & DDL Generation]
E -.->|No Direct Execution| F[Structured Metadata Artifact]
end
# Example: Injecting targeted skills into a lightweight SkillAgent
agent = SkillAgent(
skills=[
MtaTransitSkill(), # Domain-specific schema & validation rules
BigQueryGovernanceSkill() # Target warehouse conventions & partitioning rules
]
)
Domain Skills vs. Platform Governance Skills: A domain skill (likeMtaTransitSkill) defines business entities, field requirements, and expected schema contracts. A platform governance skill (likeBigQueryGovernanceSkill) defines engine-specific formatting standards—such as snake_case column conventions, timestamp parsing constraints, and lakehouse partitioning rules. Neither skill executes database queries; they purely supply the declarative rules the agent needs to synthesize compliant DDL.
By isolating domain and platform rules into distinct skills, teams can refine schema requirements and governance guidelines without modifying the underlying agent execution engine.
A Command represents deterministic execution. It answers the question: What can the agent do?
In production data engineering, data warehouse modifications, bucket operations, and network transactions must never be left to stochastic LLM text generation. If you ask an LLM to generate and run an ALTER TABLE statement directly five times, you risk getting five subtly different variations—or worse, a hallucinated column that breaks downstream dashboards.
Commands provide absolute predictability by wrapping operations in testable, deterministic code that implements the Strategy Pattern.
graph TD
subgraph Commands Architecture
CMD[Agent Command] --> STRAT[WarehouseTableStrategy]
STRAT --> BQ[BigQuery Strategy]
STRAT --> SF[Snowflake Strategy]
STRAT --> SQL[SQL Server Strategy]
BQ --> MCP[Model Context Protocol MCP]
SF --> MCP
SQL --> MCP
end
GetFileSampleCommand, CreateExternalTableCommand). The LLM does not improvise the execution path.WarehouseTableStrategy. Whether the target is BigQuery, Snowflake, or Databricks, the agent invokes the same interface while the underlying strategy handles engine-specific semantics.routing.yaml), rather than probabilistic prompt suggestions:# routing.yaml: Deterministic ingestion targets
domains:
mta_transit:
target_warehouse: "bigquery"
dataset: "transit_staging"
strategy: "external_table_v2"
factory_telemetry:
target_warehouse: "snowflake"
database: "iot_telemetry"
To understand how skills and commands interact in production, consider a real-world scenario from the Metropolitan Transportation Authority (MTA) New York City transit turnstile data—the canonical dataset explored throughout my book, Data Engineering Process Fundamentals.
In a modern Zero-ETL lakehouse, raw transit feeds land in cloud storage buckets as CSV or Parquet files. BigQuery queries these files directly via external tables, avoiding heavy ETL ingestion jobs.
When a new file format arrives (Version 2) containing altered column names, restructured timestamps, or additional device counters, traditional automated pipelines fail catastrophically. Unmapped columns crash downstream queries, while naive auto-ingestion can pollute warehouse tables with broken types.
sequenceDiagram
autonumber
actor Lake as Data Lake (Storage Trigger)
participant Agent as SkillAgent Core (Skills & Commands)
participant Gate as Security Gate (Pre-Hook & HITL)
participant BQ as Data Warehouse (BigQuery via MCP)
Lake->>Agent: Event: New File Dropped (turnstile_v2.csv)
Note over Agent: 1. GetFileSampleCommand fetches 5-row sample<br/>2. MTA Skill detects schema drift<br/>3. Skill synthesizes normalized V2 DDL
Agent->>Gate: Dispatch CreateExternalTableCommand(DDL)
Note over Gate: Pre-Use Hook intercepts tool call<br/>Agent placed on hold (Suspended)
Gate->>Gate: Security Prompt: Approve DDL Mutation?
alt Human Approves (yes)
Gate->>BQ: Execute DDL via BigQuery MCP Tool
BQ-->>Agent: External Table Created (ext_mta_turnstile_v2 live)
Note over Agent,BQ: Lakehouse reads V2 immediately at wire speed
else Human Rejects (no)
Gate-->>Agent: Mutation Aborted (Workflow Ends Safely)
Note over BQ: Fail-Closed: Table is Never Created
end
A critical design requirement of this architecture is deterministic inspection:
GetFileSampleCommand) to extract a bounded slice of records (the header plus the first several rows).MtaTransitSkill). The skill contains canonical schema rules, column mappings, and enterprise formatting conventions. When schema drift is detected, the skill does not simply panic or halt the pipeline. Instead, it normalizes the new external table schema—mapping renamed raw columns (control_area, remote_unit, sub_device_id) to standard types and generating a compliant DDL statement for BigQuery.ext_mta_turnstile_v2), the data lakehouse can consume the new feed immediately without breaking existing V1 analytics or crashing scheduled batch jobs.While the skill has the domain intelligence to normalize schemas, autonomous AI must never possess unchecked authority to modify production database structures.
To enforce security:
CreateExternalTableCommand, the command dispatches an MCP tool call. The SecureToolAgent layer evaluates this call against registered Pre-Use Hooks. Read-only queries pass freely, but state-mutating operations (DDL/DML) are intercepted before they ever reach the database server.yes): If the human verifies the DDL and approves, the hook unblocks execution. The command forwards the query through the BigQuery MCP tool, provisioning the new table seamlessly.no): If the human rejects the change or detects an anomaly, the process halts immediately and the table is never created. This fail-closed model guarantees that hallucinations, schema corruption, or unauthorized alterations cannot reach the analytical warehouse.During the live session, we walked through this pattern using Google Antigravity and the Agent Development Kit (ADK) CLI, operating on the open-source repository.
The agent class inherits from SecureToolAgent, which equips it with standard MCP tool connectivity and security controls. We then inject domain skills and executable commands:
class SkillAgent(SecureToolAgent):
"""
Modular Agent demonstrating dynamic skill injection
and deterministic command execution.
"""
def __init__(self, skills=None, commands=None, routing_config="routing.yaml"):
super().__init__()
self.skills = skills or []
self.commands = commands or []
self.routing_config = load_routing(routing_config)
def register_skill(self, skill):
self.skills.append(skill)
# Mounts domain specifications into agent knowledge boundary
self.mount_knowledge(skill.get_manifest())
def register_command(self, command):
self.commands.append(command)
# Binds executable tool signatures to the agent runtime
self.bind_tool(command.get_tool_signature())
To trigger the agent workflow, we invoke the Agent Development Kit (ADK) CLI:
adk agent run skill-agent \
--event "storage.object.created" \
--bucket "mta-turnstile-lake" \
--path "feeds/v2/turnstile_2026_09.csv"
skill-agent: The target agent profile to instantiate. The runtime configures the agent kernel and injects its specified skills (MtaTransitSkill, BigQueryGovernanceSkill) and commands (GetFileSampleCommand, CreateExternalTableCommand).--event "storage.object.created": Specifies the incoming event type. This matches standard cloud lifecycle event definitions (e.g., Google Cloud Storage Object Finalize, AWS S3 s3:ObjectCreated, or Azure Event Grid).--bucket "mta-turnstile-lake": The target storage bucket in the data lake where the raw payload arrived.--path "feeds/v2/turnstile_2026_09.csv": The relative URI key of the newly dropped file.Production vs. Local Emulation: In a live enterprise deployment, developers do not manually trigger terminal commands. Instead, automated cloud infrastructure listens for file drops: a Cloud Storage bucket event publishes to Google Cloud Pub/Sub or Eventarc, which invokes a serverless container (such as Google Cloud Run or a Cloud Function) running the ADK agent runtime. The CLI command faithfully replicates this exact production event-driven behavior for local testing, CI/CD verification, and interactive playbook orchestration with Google Antigravity.
Upon receiving the event, the agent executes GetFileSampleCommand. Rather than streaming entire megabytes into the prompt, it fetches a bounded 5-row sample. The MTA domain skill evaluates the sample against known schemas, identifies the drift, and synthesizes a normalized DDL statement:
[INFO] Executing: GetFileSampleCommand(uri="gs://mta-turnstile-lake/feeds/v2/turnstile_2026_09.csv")
[INFO] Sample loaded: 5 rows retrieved.
[WARN] Schema Drift Detected!
Missing Baseline Columns: [C/A, UNIT, SCP]
Detected V2 Columns: [control_area, remote_unit, sub_device_id, net_entries, net_exits]
[INFO] Domain Skill generated BigQuery External Table DDL:
CREATE OR REPLACE EXTERNAL TABLE `transit_staging.ext_mta_turnstile_v2`
(
control_area STRING,
remote_unit STRING,
sub_device_id STRING,
event_timestamp TIMESTAMP,
net_entries INT64,
net_exits INT64
)
OPTIONS (
format = 'CSV',
uris = ['gs://mta-turnstile-lake/feeds/v2/*.csv'],
skip_leading_rows = 1
);
By normalizing the new external table schema to match lakehouse partitioning and naming conventions, the skill guarantees that the pipeline can consume the new file format without failing or interrupting existing workloads.
Before mutating the data warehouse, the agent initiates table provisioning via CreateExternalTableCommand. However, because the agent inherits from SecureToolAgent, the tool call is intercepted by a Pre-Use Hook.
The hook determines that creating or replacing an external table constitutes an infrastructure mutation requiring human authorization. The agent's execution is immediately suspended (put on hold), and an interactive prompt is rendered:
[SECURITY GATE] Action requires administrative authorization:
Mutation: Create BigQuery External Table 'ext_mta_turnstile_v2'
Do you approve the execution of this DDL? (yes/no): yes
yes): The operator reviews the proposed DDL and grants approval. The agent resumes execution, and the pre-use hook releases the intercepted command. The command invokes the BigQuery MCP tool, executes the DDL, and provisions ext_mta_turnstile_v2. A quick refresh of the BigQuery console confirms that the V2 table is live and queryable at wire speed.no): If the operator detects an irregularity or rejects the change, the pre-use hook terminates the workflow immediately. The process ends, and the table is never created. This fail-closed architecture ensures that no unauthorized or broken schemas can ever compromise analytical pipelines.While the live demonstration successfully created the V2 external table, provisioning a new schema is not the end of the architectural story—it represents the beginning of an ongoing, production-grade data lifecycle:
With ext_mta_turnstile_v2 successfully deployed, the data warehouse operates in a live multi-version coexistence state. Downstream analytics, scheduled dashboards, and legacy ETL pipelines reading the original V1 table continue running without downtime. New applications and modernized reports can simultaneously begin querying the V2 schema. Over time, as data producers migrate and incoming V1 storage events cease, the legacy V1 external table can be gracefully deprecated and retired without risking breaking changes.
Data feeds continuously evolve. Over time, an upstream vendor or sensor system might drop an unannounced Version 3 (v3) payload into the lakehouse. This arrival triggers the exact same event-driven loop:
turnstile_2026_v3.csv.GetFileSampleCommand to inspect the sample deterministically.Not all schema changes are simple column additions. In real-world enterprise environments, a V3 payload might introduce complex edge cases—such as deeply nested JSON structures, conflicting timestamp formats, or ambiguous domain concepts that defy existing heuristic rules.
In these scenarios, the LLM-driven skill may fail to produce an acceptable or compliant schema proposal. It might hallucinate a data type, mishandle a nested field, or violate lakehouse partitioning rules.
This failure mode is precisely why the Human-in-the-Loop (HITL) gate is indispensable:
no.MtaTransitSkill). By adding explicit domain heuristics, specifying parsing patterns for the new nested fields, or clarifying type conversion rules in the skill's Markdown specification, they close the knowledge gap.Skills and Commands as First-Class SDLC Artifacts: Creating new skills or updating existing skills and commands is not a sign of pipeline failure—it is the natural, expected lifecycle of an agentic data system. Treating skills as version-controlled specifications ensures that as data domains evolve, agent intelligence grows incrementally alongside the business.
Refactoring from a monolithic agent to modular skills and commands bridges the gap between fragile AI experiments and reliable production data systems.
| Concept | Monolithic Agent | Decoupled Skills & Commands |
|---|---|---|
| Domain Knowledge | Crammed into one prompt | Modular Markdown manifests (Skills) |
| Execution | Stochastic LLM query generation | Reusable, strategy-backed classes (Commands) |
| Token Cost | High (full prompt on every call) | Low (load only active domain and tools) |
| Portability | Hardcoded to one database | Abstracted across engines (BigQuery, Snowflake) |
| Safety & Governance | Uncontrolled side-effects | Pre-Tool Hooks & Human-in-the-Loop gates |
By establishing these boundaries, data teams can confidently harness generative AI to handle real-world data entropy while keeping pipelines robust, secure, and predictable.
Thanks for reading! 😊 If you found these architectural patterns helpful, let's connect and continue the conversation:
👉 Originally published at ozkary.com