Configuration file

New to Dataglot?. Start with the step-by-step quickstart (install → config → first query). This page is the exhaustive field reference.

The Dataglot server takes a bootstrap TOML config — bind address, TLS, auth, and any initial catalogs/policies — passed via --config <path> (or the DATAGLOT_CONFIG env var). Every block is optional; an empty file boots a bare server. Catalogs, secrets, users, roles, and policies can also be created at runtime over SQL DDL and persisted in the meta store — see runtime configuration.

Fastest start — scaffold a commented starter, then point the server at it:

dataglot init                       # writes ./dataglot.toml (refuses to clobber; --force to overwrite)
# edit dataglot.toml, export the DSN env var it names, then:
dataglot --config dataglot.toml

dataglot --print-example-config streams the same content to stdout instead (… > custom.toml, or pipe it elsewhere). The generated file uses real # comments, so it stays valid TOML you can trim down. Booting with no catalogs prints a banner telling you exactly this.

Worked example configs (exercised by the test suite) live in examples/demo:

• dataglot.toml — postgres + mysql catalogs, static masks + row filters
• dataglot-tpch.toml — TPC-H benchmark setup

Credential rule (applies everywhere): secrets are never written in the config file. Every credential field has an *_env twin naming an environment variable; the server resolves it once at boot and fails fast if it's missing. Literal-secret fields (password, token, secret_access_key, dsn with inline password) exist as dev-only escape hatches. Secrets never appear in logs, errors, or plans.

Top-level fields

Plus the nested blocks documented below: observability, catalogs, masks, row_filters, access_denials, column_grants, identities, roles, governance, derived_products, maintenance, auth, authz, pgwire_tls, rate_limit, policy_explain, lineage, governance_publishers, webhook, catalog_service, flight_sql, ballista.

observability

The same listener serves GET /lineage: the boot-time column-lineage graph across declared derived_products, as JSON ({products, nodes, edges}), with each column annotated configured / propagated when a mask covers it. This is the inspection surface for lineage-propagated governance (a mask on users.email extends to every derived column that descends from it). Loopback-only, like /metrics: it exposes table/column names and product SQL.

catalogs — data sources

A map of catalog name → source config. The name becomes the SQL catalog: SELECT * FROM <name>.<schema>.<table>. Each entry is discriminated by kind.

Declaring catalogs without a file (env vars)

For containerized / 12-factor deploys you can declare catalogs entirely via the environment — no config file needed. Each DATAGLOT_CATALOG_<NAME> variable holds a single catalog object as JSON; the <NAME> suffix is lowercased to form the catalog name:

export DATAGLOT_CATALOG_PG='{"kind":"postgres","dsn_env":"PG_DSN"}'
export DATAGLOT_CATALOG_MYSQL_DEMO='{"kind":"mysql","dsn_env":"MYSQL_DSN"}'
export PG_DSN='host=db port=5432 user=me password=... dbname=app'
export MYSQL_DSN='mysql://me:...@db:3306/app'
dataglot            # boots with catalogs `pg` and `mysql_demo`, no file

Env catalogs are merged after any --config file and override a file-declared catalog of the same name. Secrets still never live in the value directly — use the *_env twins (dsn_env, …) exactly as in the file. (Governance, identities, and other structural blocks remain file-only for now.)

kind: "postgres" / kind: "mysql"

[catalogs.pg]
kind = "postgres"
dsn_env = "DEMO_PG_DSN"
tls = "require"
tls_ca_file = "/etc/ssl/private-ca.pem"

kind: "snowflake"

[catalogs.sf]
kind = "snowflake"
account = "acme-prod1"
warehouse = "COMPUTE_WH"
database = "ANALYTICS"
user = "DATAGLOT_SVC"
password_env = "SNOWFLAKE_PASSWORD"
schema = "PUBLIC"
role = "ANALYST_ROLE"

account, warehouse, database, user are required; exactly one of password / password_env is required; schema and role are optional. Transport is HTTPS by nature of the Snowflake API.

kind: "oracle" (feature-gated: build with oracle and/or oracle-pure)

[catalogs.ora]
kind = "oracle"
dsn = "//db.internal:1521/ORCLPDB1"
user = "DATAGLOT"
password_env = "ORACLE_PASSWORD"
driver = "pure"

kind: "warehouse" — lakehouse tables (REST catalog + S3)

[catalogs.lakehouse]
kind = "warehouse"
catalog_url = "http://lakekeeper:8181/catalog"
warehouse = "demo"
s3_endpoint = "http://minio:9000"
s3_region = "us-east-1"
credentials = { kind = "static", access_key_id = "minio", secret_access_key_env = "LAKEHOUSE_S3_SECRET" }

catalog_url, warehouse, credentials are required. credentials.kind is "environment" (standard AWS env vars) or "static" (access_key_id + exactly one of secret_access_key / secret_access_key_env). s3_endpoint targets S3-compatible stores (MinIO, RustFS); omit for AWS S3.

kind: "object_storage" — query files directly

[catalogs.files]
kind = "object_storage"

[catalogs.files.s3]
endpoint = "http://minio:9000"
region = "us-east-1"
access_key_id = "minioadmin"
secret_access_key_env = "S3_SECRET"
path_style_access = true

[[catalogs.files.tables]]
name = "events"
url = "s3://lake/events/*.parquet"
format = "parquet"

[[catalogs.files.tables]]
name = "signups"
url = "file:///data/signups.csv"
format = "csv"
schema = "raw"

[[catalogs.files.tables]]
name = "logs"
url = "file:///data/logs.json"
format = "json"

Each table becomes <catalog>.<schema>.<name> (schema defaults to public).

• Formats: parquet, csv (a header row is assumed), and json (newline-delimited — one JSON object per line). Schema is inferred at boot.
• Schemes: file:// always; s3:// when the [s3] block is present. Globs work (s3://lake/events/*.parquet, file:///data/part-*.parquet).
• [s3] block: omit endpoint for real AWS; set it for S3-compatibles (MinIO, Cloudflare R2, …). secret_access_key_env names an env var holding the secret (preferred over inline secret_access_key). Omit both to fall back to the ambient AWS credential chain (for public buckets, that's fine). path_style_access defaults to true (MinIO / self-hosted); set false for virtual-hosted AWS buckets. region defaults to us-east-1.
• gs:// and abfs:// are not yet supported.

kind: "odata" / kind: "sap_s4hana" — REST sources

[catalogs.sap]
kind = "sap_s4hana"
service_url = "https://host/sap/opu/odata/sap/API_BUSINESS_PARTNER"
sap_client = "100"
sap_language = "EN"
auth = { kind = "basic", user = "svc", password_env = "SAP_PASSWORD" }

service_url and auth are required. auth.kind is "basic" (user

• one of password/password_env) or "bearer" (one of token/token_env). The sap_s4hana kind adds the optional sap_client / sap_language request headers.

kind: "rest" — generic REST/JSON sources (Salesforce, HTTP APIs)

For JSON APIs that aren't OData (Salesforce and similar). Unlike OData there's no metadata document, so each table declares its URL, where the row array lives (records_path), its columns, and how to paginate.

[catalogs.salesforce]
kind = "rest"
schema = "public"
auth = { kind = "bearer", token_env = "SF_TOKEN" }

[[catalogs.salesforce.tables]]
name = "account"
url = "https://acme.my.salesforce.com/services/data/v58.0/query?q=SELECT+Id,Name,AnnualRevenue+FROM+Account"
records_path = "records"
pagination = { kind = "next_link", next_path = "nextRecordsUrl" }
columns = [
  { name = "Id", type = "utf8" },
  { name = "Name", type = "utf8" },
  { name = "AnnualRevenue", type = "float64", nullable = true },
]

Each table becomes <catalog>.<schema>.<name> (schema defaults to public). REST/JSON field names are case-sensitive, so quote them in SQL.

• url — the request endpoint (may carry a query string, e.g. a SOQL query).
• records_path — dot-path to the row array in the response ("" = the body is itself the array; "records" for Salesforce).
• columns — one per selected field: name + type (utf8, boolean, int32, int64, float64) + optional nullable (default true).
• pagination — { "kind": "none" } (default) or { "kind": "next_link", "next_path": "<dot-path to next URL>" } to follow a next-page link (absolute or relative) until absent — e.g. Salesforce's nextRecordsUrl.
• auth — "none" (default), "basic" (user + password/password_env), "bearer" (token/token_env), "header" (name + value/value_env for an API-key header), or "oauth2" (see below). Secrets prefer the *_env form.

Note that REST catalogs are single-node only: in distributed mode a query against one is refused with a "run single-node" message (no distributed plan codec).

For Salesforce and other OAuth 2.0 client-credentials sources, the connector acquires and refreshes its own bearer token — no static token to rotate:

[catalogs.salesforce.auth]
kind = "oauth2"
token_url = "https://login.salesforce.com/services/oauth2/token"
client_id_env = "SF_CLIENT_ID"
client_secret_env = "SF_CLIENT_SECRET"
scope = "api"

token_url is required; client_id and client_secret each take exactly one of the literal or *_env form; scope is optional. OAuth2 is connector-level (one refreshed token serves every table), and the token is fetched lazily on first query, cached, and refreshed before expiry.

Like OData, REST is a direct TableProvider — it federates and is governed (masks / row-filters apply) exactly like a SQL source.

Governance and policy

Note. Governance (masks, row filters, access-deny) is the third access-control layer, after authentication and GRANT authorization. For how the three compose — and why a superuser bypasses grants but never masks — see access control.

masks — column masking

[[masks]]
table = "users"
column = "email"
mask_literal = "***@example.com"

[[masks]]
table = "pg.public.users"
column = "ssn"
mask_type = { kind = "show_last", keep = 4 }
priority = 10

row_filters — row-level filtering

[[row_filters]]
table = "users"
predicate = { kind = "eq_string", column = "email", value = "bob@example.com" }

[[row_filters]]
table = "orders"
predicate = { kind = "sql", sql = "region = 'EU' AND status IS NOT NULL" }

Predicate kinds: eq_string, eq_int, gt_int (declarative, typed), or sql (arbitrary boolean expression, parsed at boot — for non-Utf8 columns prefer the typed variants or explicit CASTs). Filters evaluate on unmasked values and wrap the table scan — no bypass.

access_denials — table/column deny

[[access_denials]]
table = "salaries"
groups = ["analyst"]

[[access_denials]]
table = "users"
column = "ssn"
groups = []

Enforced plan-time before masking; a denied query fails with permission denied. Empty groups denies everyone.

column_grants — column whitelists

The inverse of access_denials: instead of naming what to hide, name the only columns a group may see.

[[column_grants]]
table   = "pg.public.employees"
columns = ["id", "name", "department"]   # the only visible columns
groups  = ["ops-analyst"]                # for these groups
# org   = "acme"                         # optionally tenant-scoped

Non-whitelisted columns are projected away — absent from the result entirely, not masked and not null. This is enforced at the analyzer stage rather than the optimizer, because dropping a column changes the plan's output schema.

Multiple grants on one table are additive — the visible set is their union. A table with no applicable grant is unrestricted, so a whitelist only constrains tables it names.

identities and roles

[identities.alice]
org = "acme"
groups = ["analyst"]
password_env = "ALICE_PASSWORD"

[roles.pii_reader]
users = ["alice"]
groups = ["auditor"]

identities maps the pgwire username to org + group memberships (the policy identity; unknown usernames get empty groups). password_env is consulted only under auth.mode = "md5". A roles entry folds into the session's effective groups when its user or any group matches.

governance — tag-based policies

[governance]
tags = [ { id = "pii", org = "acme", name = "PII" } ]
policies = [ { id = "mask-pii", org = "acme", tag = "pii", group = "analyst", rule = { kind = "mask", mask_literal = "***" } } ]
columns = [ { table = "users", column = "email", tags = ["pii"] } ]

Tag → policy → column indirection: tag a column and every policy attached to that tag fires for sessions in the policy's group. Rule kinds mirror the static blocks: mask (+mask_literal) and row_filter (+predicate). The inbound governance webhook (below) mutates this registry at runtime.

derived_products — data products with lineage

[[derived_products]]
name = "eu_revenue"
sql = "SELECT ... FROM orders ..."
backing = "materialized"
materialization = { warehouse = "lakehouse", namespace = "products", refresh_every = "15m" }

Planned once at boot to extract column lineage, so masks on source columns propagate to derived columns. backing is "live" (default; planned per read) or "materialized" (refreshed into a warehouse table on the refresh_every cadence — durations like "30s", "15m", "1h", "2d").

A materialized product whose sql federates across sources is the no-dblink migration pattern: a query that on a legacy stack would be an Oracle dblink + CREATE TABLE job becomes a governed federated read persisted to the lakehouse on a schedule — no per-source link, no data-movement script.

Refresh status for every materialized product is observable at GET /api/materialization (and the dashboard's Materialization tab): per product — state (pending/running/success/error), last row count and duration, when it last ran, the approximate next run, and a run/failure tally. A failed refresh is non-fatal (the prior snapshot is retained and the scheduler retries next tick); its redacted error is surfaced there. Loopback-only, same posture as the rest of /api.

Security

auth

jwt mode reads an [auth.jwt] block (algorithm hs256/rs256/es256, secret_env or public_key_file, groups_claim, issuer, audience, leeway_secs); ldap reads [auth.ldap]. See authentication for the mode details.

authz — object authorization (GRANT)

Grants are written with the GRANT / REVOKE DDL, apply per role or user, and are org-scoped. Superuser sessions bypass grant enforcement (but never column masks or row filters). See authentication → Authorization, and access control for how authentication, GRANT, and governance compose.

pgwire_tls — client↔server encryption

[pgwire_tls]
cert_file = "/etc/tls/server.pem"
key_file = "/etc/tls/server.key"
mode = "require"

mode is "prefer" (default — accept TLS and plaintext) or "require" (reject plaintext). Omitting the block leaves the listener plaintext (and md5 auth then warns at boot).

rate_limit — connection admission

All optional; omitted = unlimited. Rejections emit dataglot::audit events and bump dataglot_pgwire_connections_rejected_total{reason}.

These ceilings (plus memory_limit_bytes) and the live usage against them — active connections, the busiest IP / identity bucket, and cumulative rejections by reason — are served at GET /api/limits and rendered as the dashboard's Resource limits panel (Sessions tab), so an operator can see headroom at a glance. Loopback-only, same posture as the rest of /api.

policy_explain — explainability endpoint

[policy_explain]
addr = "127.0.0.1:8085"
token_env = "POLICY_EXPLAIN_TOKEN"

Enables POST /policy/explain: plans a SQL string (never executes) and reports the mask / row-filter / deny decisions for a given identity. token_env names a bearer token; unset ⇒ open endpoint + boot warning.

Integrations

lineage — OpenLineage emitter

[lineage]
kind = "openlineage_http"
endpoint = "http://marquez:5000/api/v1/lineage"
namespace = "dataglot.acme"

Emits an OpenLineage RunEvent (with column-level columnLineage facets) per query. Compatible with DataHub, Marquez, OpenMetadata, Gravitino, Informatica. Omit for no emission.

governance_publishers — outbound metadata

[[governance_publishers]]
kind = "datahub"
gms_endpoint = "http://datahub-gms:8080"
bearer_token_env = "DATAHUB_TOKEN"

Publishes data products / column metadata to the platform at boot and on binding changes. bearer_token_env optional (local DataHub dev deployments run unauthenticated).

webhook — inbound governance (policy ingestion)

[webhook]
addr = "0.0.0.0:8084"
secret_env = "DATAGLOT_WEBHOOK_SECRET"

HMAC-SHA256-authenticated endpoint receiving tag/policy/certification events from a governance platform's actions framework; propagates into enforcement in under 60 seconds. Both fields required to enable.

catalog_service

The meta store that persists runtime control-plane DDL (CREATE CATALOG / SECRET / USER / ROLE / MASK …). Two backends — pick by the keys you set:

[catalog_service]
path = "/var/lib/dataglot/meta.redb"
org_id = "default"

[catalog_service]
dsn = "postgres://user:pass@host/catalog"
org_id = "default"

Omitted = catalogs come only from catalogs (file/env) and no runtime DDL is available. When configured, the store is also a source of truth for catalogs: at boot the server unions the file/env config with the source configs stored in the meta store; for a name the file also declares, the file wins. Stored source configs are credential-free — they name *_env vars or dsn_secret references, never secret values; encrypting secrets at rest needs DATAGLOT_SECRET_KEY. See runtime configuration for the runtime DDL + secrets detail.

maintenance — scheduled compaction

[[maintenance.compaction]]
warehouse = "lakehouse"
namespace = "products"
table = "eu_revenue"
compact_every = "6h"

[[maintenance.orphan_cleanup]]
warehouse = "lakehouse"
namespace = "products"
sweep_every = "1h"
min_age = "6h"

compaction rewrites a table into fewer, larger files (the classic lakehouse OPTIMIZE); orphan_cleanup sweeps stale staging/parked tables left by interrupted writes. Both run on the in-process scheduler. Their live status — per job: state, last run, files affected / tables swept, next run, and any redacted error — is served at GET /api/maintenance and rendered as the Warehouse maintenance panel on the dashboard's Materialization tab. Loopback-only, same posture as the rest of /api.

flight_sql — Arrow Flight SQL endpoint (feature-gated: flight_sql)

A second query interface alongside pg-wire, speaking Arrow Flight SQL over gRPC — native Arrow out, no row conversion. It shares the same SessionContext and the same plan-time policy enforcement, so masks, row filters, denials, and grants apply identically on both interfaces.

[flight_sql]
addr = "0.0.0.0:32010"

[flight_sql.tls]
cert_file = "/etc/dataglot/flight.crt"
key_file  = "/etc/dataglot/flight.key"

Omit the tls block for plaintext gRPC (h2c). Certificates are referenced by path and never inlined. The block requires a build with the flight_sql feature — it is not in the published binaries or container image, so see building from source.

ballista — distributed execution (feature-gated: ballista)

[ballista]
standalone_parallelism = 4
rest_api_port = 50050

Spins up Dataglot's in-process distributed cluster (1 scheduler + executors with the given task slots). Present-but-uncompiled is a boot error.

Distributed-capable catalogs: federated SQL sources (postgres, mysql, snowflake), Iceberg warehouses (warehouse — lazily rebuilt on executors, the catalog load_table happens at execution time), and object_storage files. The direct-TableProvider sources (odata, sap_s4hana, rest) always run single-node, and oracle / adbc distribute only when the server and the executors were built with their feature. Anything else fails with a clear "not available in distributed mode" error and should be queried single-node.

rest_api_port serves the scheduler's observability REST API on loopback (/api/state, /api/executors, /api/jobs, /api/job/{id}/stages, /api/job/{id}/dot and /dot_svg execution graphs — SVG requires graphviz dot on the host) — the data source for live cluster monitoring. Default 50050; null disables. Loopback-only by design: the endpoints are unauthenticated and can expose query text.

The same listener serves cluster metrics in Prometheus format at /api/metrics — job/task/executor counters from the cluster scheduler. This is a second scrape target alongside the server's own metrics_addr (:9090/metrics, pgwire/session metrics):

# prometheus.yml
scrape_configs:
  - job_name: dataglot-server
    metrics_path: /metrics
    static_configs: [{ targets: ["localhost:9090"] }]
  - job_name: dataglot-cluster          # only when ballista is enabled
    metrics_path: /api/metrics
    static_configs: [{ targets: ["localhost:50050"] }]

Since both listeners are loopback-only, run the Prometheus scraper on the same host (or bridge with a local agent).

CLI flags and environment variables

Every flag has an env-var twin; precedence is CLI > env > config file > built-in default.

Secret-bearing env vars are whatever names your config's *_env fields declare — the server reads each once at boot and refuses to start if one is missing.

Feature gates

The stock dataglot-server binary compiles the common connectors in unconditionally — kind values postgres / mysql / warehouse / snowflake / odata / rest all work out of the box. (Feature gates apply at the dataglot-federation library level, where all = postgres, mysql, iceberg, odata, rest; iceberg backs the user-facing kind: "warehouse".) The connectors the server leaves opt-in:

A server built without a needed feature rejects the corresponding config at boot with a clear, credential-free error.