Configuration
Copy config.example.yaml in the repository root and adjust for your environment.
queryflux:
externalAddress: http://localhost:8080
frontends:
trinoHttp:
enabled: true
port: 8080
snowflakeHttp:
enabled: true
port: 8445
persistence:
type: inMemory # or: postgres
clusters:
trino-1:
engine: trino
endpoint: http://trino-host:8080
enabled: true
auth:
type: basic
username: user
password: pass
duckdb-1:
engine: duckDb
enabled: true
databasePath: /tmp/queryflux.duckdb
clusterGroups:
trino-default:
maxRunningQueries: 100
members: [trino-1]
duckdb-local:
maxRunningQueries: 4
members: [duckdb-1]
routers:
- type: protocolBased
trinoHttp: trino-default
snowflakeHttp: trino-default
snowflakeSqlApi: trino-default
- type: header
headerName: x-target-engine
headerValueToGroup:
duckdb: duckdb-local
routingFallback: trino-default
SQL translation
translation:
mode: bestEffort # strict rejects required translation when unavailable or failed
errorOnUnsupported: false
bestEffort preserves the original SQL on translation failure. Use strict to reject
the query before backend submission instead. Compatible dialects without fixups bypass
translation; dialect-only fallback remains allowed when it succeeds. The legacy
errorOnUnsupported: true option also enables strict behavior. Restart QueryFlux after
changing this policy. See Query translation for
fallback counters and query history outcomes.
translation:
cache:
translations: 4096 # translated SQL; 0 disables
tableExtraction: 4096 # access-control table extraction and schema table references
fingerprints: 4096 # query-history fingerprints
Repeated queries reuse earlier results instead of parsing again. Each cache is a bounded LRU keyed on everything that changes its result (SQL text, dialects, fixup scripts, schema, strict mode), so it needs no flush on reload. Set a size to 0 to disable that cache. These settings are local to each process, including workers. See Query translation.
Authentication, authorization & backend identity
auth:
provider: oidc # none | static | oidc | ldap
required: true
oidc:
issuer: https://keycloak.internal/realms/my-realm
jwksUri: https://keycloak.internal/realms/my-realm/protocol/openid-connect/certs
audience: queryflux
authorization:
provider: none # none | openfga
clusters:
trino-1:
engine: trino
endpoint: https://trino.internal:8443
auth: # Type 1 — QueryFlux's own service credential
type: basic
username: qf_svc
password: "..."
queryAuth:
type: impersonate # serviceAccount | passthrough | impersonate | tokenExchange
auth controls who can authenticate to QueryFlux; authorization controls which cluster groups they can route to; clusters[].queryAuth controls which identity the query runs as on the backend engine — independent of the client's own credential. See Authentication & identity for the full picture, a decision guide for queryAuth modes, and per-engine setup requirements.
Access control
accessControl:
enabled: true # default for cluster groups; false = opt-in per group
defaultConnection: prod # groups without an override use this connection
connections:
prod:
provider: opa
opa:
url: http://localhost:8181
decisionPath: /v1/data/queryflux/access
timeoutMs: 1000
operations: [table.select]
onMissingSchema: evaluate
failOpen: false
cacheTtlMs: 5000
sessionParamKeys: [customer_id, tenant_id]
groups:
sandbox:
enabled: false
When set, QueryFlux connects to OPA and asks whether the verified identity may read each referenced table, then optionally rewrites SQL with row filters and column masks before dialect translation. QueryFlux does not author the policy (Rego stays in OPA).
enabled— global default for cluster groups;falsemeans opt-in only wheregroups.<name>.enabled: true.connections— named policy-provider connections. No name is reserved.defaultConnection— which connection a group uses when it has no explicitgroups.<name>.connectionoverride. Unset means such a group gets no access control at all.groups.<name>.enabled— per cluster group: inherit (omit), force on, or skip OPA (false).groups.<name>.connection— route this group to a different named connection instead ofdefaultConnection(network segmentation, blast-radius isolation, provider migration — see Multiple connections).- Studio — the Access Control page can add/edit/remove any number of connections and set
defaultConnection+ per-group scope.
Omit the block (or { "enabled": false } with no connections, via the Admin API) to disconnect. Full reference: Access control and OPA provider.
Admin API
queryflux:
adminApi:
port: 9000 # Admin REST API + Studio proxy port (default: 9000)
username: admin # Bootstrap admin username — see note below (default: admin)
password: admin # Bootstrap admin password — see note below (default: admin)
username and password are the bootstrap credentials used on first boot. After you change the password through Studio's Security page, the new bcrypt hash is stored in Postgres and the YAML values are ignored.
Environment variables QUERYFLUX_ADMIN_USER and QUERYFLUX_ADMIN_PASSWORD override the YAML fields and follow the same bootstrap semantics.
See Studio & Admin Auth for the full credential priority rules and password-change instructions.
Persistence and distributed mode
queryflux:
persistence:
type: postgres
postgres:
url: postgres://queryflux:queryflux@localhost:5433/queryflux
# Optional — defaults to true when postgres persistence is configured
distributed: true
configReloadIntervalSecs: 30
With Postgres persistence and distributed: true (the default when Postgres is configured), multiple QueryFlux replicas coordinate through Postgres:
| Concern | Mechanism |
|---|---|
| Config hot-reload | configReloadIntervalSecs + immediate reload on Admin API writes |
Fleet-wide maxRunningQueries | Capacity leases in cluster_capacity_leases, counted in cluster_capacity_counters.leased; see Capacity and throughput |
| Engine running counts | Reconcile sweep publishes to cluster_capacity_counters.running |
| Queued query dispatch | Claim columns on queued_queries — one replica per query |
Set distributed: false to run Postgres-backed persistence without cross-replica coordination (single replica or explicit opt-out).
Capacity quotas. By default every admission takes a per-query lease (one call into Postgres or the control plane). With quotas, each process gets a share of every cluster's limit and admits within it locally. Set it on standalone replicas or the control plane; workers follow their control plane.
queryflux:
capacity:
mode: quotas # leases (default) | quotas
reserveFraction: 0.15 # share of each limit kept for per-query leases
quotaLeaseSecs: 15 # how long a process may admit on its quota without reporting
reportIntervalMs: 1000 # how often each process reports demand
Turn it on only once every standalone or control-plane replica runs a version with quotas. How it works: Capacity and throughput.
One database, many deployments. Set persistence.schema to keep a deployment's tables in their own schema, so several deployments (for example one per tenant) can share one database:
queryflux:
persistence:
type: postgres
url: postgres://qf:secret@db:5432/queryflux
schema: tenant_acme # lowercase identifier; created by migrations if missing
Every connection gets the schema as its search_path; migrations run in it and keep their own ledger there; startup stops if a connection resolves to another schema (for example a role whose own search_path overrides it). Postgres advisory locks and LISTEN/NOTIFY channels are database-wide, so in a pinned schema QueryFlux scopes its lock keys and channel names to the schema: two deployments never wait on each other's sweeps or capacity locks, or wake on each other's config changes. Without schema, nothing changes.
Connection pools. Each process opens three pools: query state (queryPoolSize, default 7), coordination for capacity leases, queue claims and sweep locks (coordinationPoolSize, default 8), and admin (adminPoolSize, default 2), plus one LISTEN connection. Connections open on demand. Size coordinationPoolSize to the admissions per second you expect × about 1 ms. The legacy poolSize instead sets one budget split 60/20/20, and is ignored when any of the three is set.
Query history retention. queryHistoryRetentionDays deletes query records, digest stats and cluster snapshots older than that many days. With Postgres it runs hourly, on every replica (deleting the same rows twice is harmless), in batches of 10,000 rows, each its own short transaction. A large backlog, such as after turning retention on for an existing database, is worked through over successive batches. Each statement stays short, and autovacuum keeps up. Omit it to keep history forever.
queryflux:
queryHistoryRetentionDays: 30
Helm / Kubernetes: see Run multiple replicas. Full behavior: Cluster variants, health checks & reconciliation.
Worker mode
queryflux.role selects how a process gets its config: standalone (the default), controlPlane, worker, or hub (relays a remote control plane to the workers at one site; see Hub). A worker pulls its config, capacity leases and history endpoint from a control plane, and never connects to Postgres:
queryflux:
role: worker
workerSecret: <secret>
worker:
upstream:
url: https://queryflux-control:9000
coordination: { mode: controlPlane, onUnavailable: failClosed }
A worker and whatever serves it share queryflux.workerSecret. Each role other than standalone has its own section: queryflux.controlPlane (history.endpoint, history.redaction), queryflux.worker and queryflux.hub. Every key for each role: Configuration by role. How config, capacity and history flow: Worker mode.
Query Cache
QueryFlux can cache deterministic query results to avoid repeated backend roundtrips. See the dedicated Caching page for full documentation.
Quick example:
queryflux:
cacheBackend:
scheme: s3
compression: lz4
options:
bucket: queryflux-cache
endpoint: http://localhost:19000
region: us-east-1
access_key_id: rustfs-root-user
secret_access_key: rustfs-root-password
clusterGroups:
analytics:
members: [trino-1]
maxRunningQueries: 50
cache:
enabled: true
ttlSecs: 300
maxEntrySizeMb: 64
config.example.yaml, config.local.yaml, and the serde types in queryflux-core (config.rs) are the authoritative reference. For routing semantics and clusterGroups, see Routing and clusters.
Snowflake (native connector)
engine: snowflake talks to Snowflake through the Rust snowflake-connector-rs client, so no ADBC driver has to be installed. The ADBC path (engine: adbc, driver: snowflake) still works and is the one to use for tokenExchange or passthrough query auth.
clusters:
snowflake-1:
engine: snowflake
account: myorg-myaccount
warehouse: COMPUTE_WH
role: ANALYST
database: ANALYTICS
schema: PUBLIC
auth:
type: keyPair
username: SVC_QUERYFLUX
privateKeyPem: "-----BEGIN PRIVATE KEY-----\n..."
| Key | Required | Notes |
|---|---|---|
account | yes | Account identifier (orgname-accountname). |
auth | yes | keyPair (username, privateKeyPem, optional privateKeyPassphrase) or basic (username, password). Snowflake enforces MFA on password logins, so use keyPair for service accounts. |
warehouse, role, database, schema | no | Session defaults. With warehouse set, health checks and running-query counts use SHOW WAREHOUSES, which never resumes a suspended warehouse. Without it, the health check runs SELECT 1. |
endpoint | no | Base URL override, for example a PrivateLink host. Defaults to https://<account>.snowflakecomputing.com. |
maxWaitSecs | no | How long to wait for a query response before cancelling the query. Defaults to 600. |
All queries share one service-account session, so only queryAuth: serviceAccount is supported. Variants work with a warehouse override, the same way as in the example below.
With Postgres persistence, the YAML seed doesn't store privateKeyPem. Create keyPair clusters through the Admin API (authType: keyPair, authUsername, authPassword set to the PEM, authToken set to the passphrase) or use basic auth.
Cluster variants (multi-warehouse)
ADBC clusters (engine: adbc) need their driver installed. The QueryFlux image ships none: declare them with QUERYFLUX_ADBC_DRIVERS (see ADBC drivers).
A single cluster config can define variants: named overrides that expand into separate runtime clusters at load time. Use this when one credential set targets multiple Snowflake warehouses, Databricks SQL warehouses, BigQuery projects, or similar.
clusters:
my-snowflake:
engine: adbc
driver: snowflake
uri: svc_user@myaccount/mydb/myschema
auth:
type: keyPair
username: SVC_ACCOUNT
privateKeyPem: "..."
# Optional — leave empty to use built-in Snowflake introspection
healthCheckQuery: "SHOW WAREHOUSES LIKE '{{sub_resource}}'"
variants:
- name: analytics
overrides:
warehouse: ANALYTICS_WH
- name: etl
overrides:
warehouse: ETL_WH
maxRunningQueries: 5
clusterGroups:
snowflake-pool:
maxRunningQueries: 20
members:
- my-snowflake::analytics
- my-snowflake::etl
Rules:
- Runtime names are
{base}::{variant_name}(for examplemy-snowflake::analytics). - A cluster with variants does not create a runtime cluster for the base name alone.
healthCheckQueryandreconcileQueryare set on the base config;{{sub_resource}}is substituted per variant.
Full details: Cluster variants, health checks & reconciliation.