Add a data source

Adding a source means creating a catalog: a named registration that tells Dataglot how to reach the source and what to call it. Nothing is copied, mirrored, or migrated — the source keeps its data and you get a name you can query.

There are three ways to register one. They produce the same thing.

1. Decide how the credential gets in

Do this first, because it changes the shape of the statement you write. The rule everywhere in Dataglot: a secret never belongs in a config file.

Non-DSN connectors follow the same pattern with password_env, token_env, secret_access_key_env, and so on.

To use the third form, create the secret first. It is encrypted before it reaches the meta store, so the server needs an envelope key in DATAGLOT_SECRET_KEY:

export DATAGLOT_SECRET_KEY="$(openssl rand -base64 32)"
dataglot --config bootstrap.toml

CREATE SECRET app_pg_dsn AS 'host=db port=5432 user=svc password=hunter2 dbname=app';

Only the reference is stored on the catalog; the value is decrypted at connect time and is redacted from logs and errors.

2. Create the catalog

CREATE CATALOG pg WITH (
  kind      = 'postgres',
  dsn_secret = 'app_pg_dsn'
);

[catalogs.pg]
kind = "postgres"
dsn_env = "APP_PG_DSN"
tls = "require"

The name you pick (pg) becomes the first part of every table name in that source: pg.public.users.

Sources that need nested config

object_storage, rest, and warehouse need structures rather than flat strings — a list of tables, a credentials object. Over SQL, pass those as a quoted JSON string; any option value starting with [ or { is parsed as JSON:

CREATE CATALOG files WITH (
  kind   = 'object_storage',
  tables = '[{"name":"events","url":"s3://lake/events/*.parquet","format":"parquet"}]'
);

In a config file the same thing is ordinary TOML:

[catalogs.files]
kind = "object_storage"

[[catalogs.files.tables]]
name = "events"
url = "s3://lake/events/*.parquet"
format = "parquet"

Every option of every kind is in the configuration reference; which kind you want is in data sources.

3. Verify it

Query it immediately — in the same session, no restart:

\dn                                  -- schemas Dataglot found in the source
\dt pg.public.*                      -- tables in one of them
SELECT * FROM pg.public.users LIMIT 10;

Then confirm the work is actually happening in the source rather than being dragged across the wire:

EXPLAIN FEDERATION
SELECT user_id, SUM(amount) FROM pg.public.orders GROUP BY user_id;

You should see the aggregation in the SQL shipped to pg. If you instead see a bare scan, the query is pulling rows back to be aggregated locally — usually a sign the plan couldn't be expressed in the source's dialect.

Finally, join it to something you already had. This is the point of the exercise:

SELECT u.email, s.segment
FROM   pg.public.users u
JOIN   files.public.segments s ON s.user_id = u.id;

4. Govern it before anyone else connects

A new catalog is new exposure. Masks and row filters are enforced in the query plan, so applying them is a two-line job and there is no way around them afterwards:

CREATE MASK email_mask ON pg.public.users (email) AS '***@example.com';
CREATE ROW FILTER active_only ON pg.public.users USING (active = true);

If the server runs with authz.mode = grant, the catalog is deny-unless-granted until you say otherwise:

GRANT USAGE  ON CATALOG pg      TO analyst;   -- reference the catalog
GRANT SELECT ON pg.public.users TO analyst;   -- read one table

See access control & policies for the full model.

5. Change or remove it

ALTER CATALOG pg WITH (kind = 'postgres', dsn_secret = 'app_pg_dsn_v2');
DROP  CATALOG IF EXISTS pg;

ALTER CATALOG replaces the whole option set and rebuilds the connector — always repeat kind and every option you still want. Because it rebuilds, it is also how you pick up tables added to the source since the catalog was created. DROP CATALOG removes the registration only; the source is untouched.

When it doesn't work

Where next

• Catalogs, schemas & tables — why the schema part is what it is.
• Runtime configuration — the rest of the SQL DDL surface.
• Access control & policies — govern the new source properly.