Batch pipelines keep the same shape (read, transform, validate, write) and each one is rewritten as code anyway. Answering "what does this job actually do?" then means reading a PySpark diff, and the answer cannot be diffed, linted, generated or drawn.

A pipeline is one JSON contract, executed by the framework and parsed by nothing else. It runs as-is from Python or from the CLI, and the same file opens on the Studio canvas as connected nodes.

{
  "name": "customers_curated",
  "input": { "format": "csv", "path": "/data/landing/customers" },
  "transformations": [
    { "type": "filter", "condition": "status = 'active'" },
    { "type": "cast", "columns": { "created_at": "timestamp" } },
    { "type": "drop_duplicates", "columns": ["id"] }
  ],
  "validations": {
    "on_failure": "warn",
    "rules": [
      { "type": "not_null", "columns": ["id"] },
      { "type": "unique", "columns": ["id"] }
    ],
    "report": { "format": "csv", "path": "/dq/customers", "mode": "append" }
  },
  "output": {
    "format": "parquet",
    "path": "/data/curated/customers",
    "mode": "overwrite",
    "partition_by": ["created_at"]
  }
}

Expand

The parts people drop back into code for have a declarative form: {param} template parameters resolved before parsing, {{var}} runtime variables that push a collected key set into a later read as a literal IN (...) (the declarative version of the collect() + isin() trick that makes Delta data skipping work), reusable $include fragments, and several destinations from a single pass.

{ "type": "checkpoint" },
{ "type": "collect", "column": "id_cliente", "as": "clientes_ativos" },
{ "type": "join",
  "input": { "format": "delta", "path": "vendas.bronze_eventos_cliente" },
  "with_transformations": [
    { "type": "filter", "condition": "id_cliente IN ({{clientes_ativos}})" },
    { "type": "distinct" }
  ],
  "on": "id_cliente", "how": "left" }

Studio is a browser canvas that reads and writes exactly those documents: it lints as you type, generates a pipeline with an LLM of your choice, and executes through a local runner bound to 127.0.0.1: no account, no server, no telemetry. It never invents syntax the framework does not support, and the framework never needs it to run.

Data quality is the same story: it is sparquet-cola, a standalone library, declared as a dependency and re-exported under sparquet.validation.*.

from sparquet import Sparquet

fw = Sparquet(spark={"app_name": "MyJob", "master": "local[*]"})
result = fw.run("pipeline.json", params={"dt_ref": "2026-01-01"})
print(result.summary())
fw.stop()

# or, without writing any Python at all:
# $ sparquet pipeline.json

Expand

• Locally, on Databricks, EMR, Dataproc or Synapse: the session manager detects the environment
• Python 3.9 through 3.14, on PySpark 3.4+
• Extensible: register your own readers, writers, transformations and validators
• Documented at sparquet.dev in English, Portuguese and Spanish