Monish Gosar
Back to projects

ETL Skills — MCP Toolkit

MCP toolkit that lets any agent harness read Databricks jobs, runs, schemas, and failure logs to diagnose a broken pipeline. Read-only by default, raw SQL rejected.

TypeScriptMCPDatabricks

The problem

When an ETL pipeline breaks, the useful context is scattered across job definitions, run output, schemas, and failure logs. An agent can only help debug it if it can read all of that.

But handing an agent a Databricks token is dangerous: an agent that can run arbitrary SQL can damage the warehouse it was asked to debug.

Architecture

Any MCP client — Claude Code, Codex, Cursor, Pi, or DeepSeek — talks to etl-agent-tools, a read-only stdio server. The tools are fixed, output is capped, and secrets are scrubbed before a response reaches the model. Portable skills sit on top: diagnose a pipeline, detect schema drift, propose a safe repair.

The server reads Databricks with a personal access token or an OAuth machine-to-machine service principal, dbt from run_results and manifest artifacts, and Airflow through its stable REST API. What comes back is a root-cause brief. This complements Databricks managed MCP servers rather than replacing them.

v0.3 shipped on 2026-10-14 with dbt, Airflow, Unity Catalog context, and OAuth service principals. The npm package is still v0.2.0-beta.1, Databricks inspection.

MCP clients

MCP server

Providers

Output

Select a step. An MCP client talks to the read-only server, the server reads a provider, and the agent returns a root-cause brief.

Complements Databricks managed MCP servers. Those stay responsible for Genie, ad hoc SQL, and platform development.

How it was built

Safety is the design constraint. The server is read-only by default and output is capped. In Pi, the four write actions (query, pipeline, run, repair) each require interactive confirmation, and raw SQL is rejected outright.

A guided setup command installs the skills for the chosen harness, writes the project config, and records environment-variable names without ever asking for or storing a token. Shipped as an npm beta.

Releases

  1. v0.3

    Shipped
  2. v0.2.0-beta.1

    Currentnpm

    Databricks inspection, diagnose-pipeline, schema-drift, safe-repair skills, guided installer.

What’s next

  • Snowflake is specced and not implemented.
  • Guarded mutations beyond Pi, so Claude Code and Codex can propose and confirm repairs too.