xarray-sql¶

`xarray_sql` ¶

`XarrayContext = Context` `module-attribute` ¶

The original name of Context, kept for compatibility.

`all = ['cftime', 'Context', 'XarrayContext', 'FlightSQLServer', 'read_xarray_table', 'read_xarray', 'arrow_dataset', 'arrow_datasets', 'bbox_conjuncts', 'register', 'serve', 'to_dataset', 'from_map']` `module-attribute` ¶

`Context` ¶

Bases: SessionContext

A datafusion SessionContext that also supports xarray.Datasets.

Also available under its original name, XarrayContext.

`from_dataset(name, input_table, *, table_names=None, chunks=None)` ¶

Register an xarray Dataset as one or more queryable SQL tables.

When all data variables share the same dimensions, the dataset is registered as a single table named name. When variables have differing dimensions (e.g. some on a 3D grid and others on a 4D grid), the dataset is split into one table per dimension group. The tables are registered under a SQL schema (namespace) named name and named <dim1>_<dim2>_... by default::

ctx.from_dataset('era5', ds, chunks={'time': 24})
# registers tables: 'era5.time_lat_lon' and
#                   'era5.time_lat_lon_level'
ctx.sql('SELECT AVG(temperature_2m) FROM era5.time_lat_lon')

Use table_names to override the name for specific dimension tuples::

ctx.from_dataset(
    'era5', ds,
    table_names={('time', 'lat', 'lon'): 'surface'},
)
ctx.sql('SELECT * FROM era5.surface')

For datasets with non-Gregorian cftime coordinates (e.g. 360_day, julian), a cftime() scalar UDF is automatically registered so you can write ergonomic SQL filters::

ctx.from_dataset("ds360", ds, chunks={"time": 6})
ctx.sql("SELECT * FROM ds360 WHERE time >= cftime('2000-07-01')")

.. note::

Only one ``cftime()`` UDF is registered per context, using the
units and calendar of the *first* non-Gregorian coordinate
encountered. If you register multiple datasets with *different*
non-Gregorian calendars (e.g. one 360_day and one julian), the
UDF from the first registration will be used for all subsequent
``cftime()`` calls and may produce incorrect offsets for the
other dataset. In that case, create a separate ``Context``
for each calendar.

Parameters:

Name	Type	Description	Default
`name`	`str`	The SQL identifier under which the dataset is registered. For datasets with uniform dimensions, this is the table name. For datasets with mixed dimensions, this is the name of a SQL schema (namespace) containing one table per dimension group.	required
`input_table`	`Dataset`	An xarray Dataset.	required
`table_names`	`TableNames`	Optional mapping from dimension tuples to custom table names within the schema, used when the dataset has variables with differing dimensions.	`None`
`chunks`	`Chunks`	Xarray-like chunks specification. If not provided, uses the Dataset's existing chunks, or for an unchunked Dataset its storage chunks (see register).	`None`

Returns:

Type	Description
	self, to allow chaining.

`sql(query, *args, **kwargs)` ¶

Run a SQL query, returning an XarrayDataFrame wrapper.

Identical to datafusion.SessionContext.sql except the returned object wraps the DataFusion DataFrame. The wrapper exposes .to_pandas() (unchanged), forwards every other DataFusion method via __getattr__, and adds .to_dataset(dimension_columns=[...]) for round-tripping the result back to an xr.Dataset.

Parameters:

Name	Type	Description	Default
`query`	`str`	A SQL query string.	required
`*args`		Forwarded to `SessionContext.sql`.	`()`
`**kwargs`		Forwarded to `SessionContext.sql`.	`{}`

Returns:

Type	Description
`XarrayDataFrame`	An XarrayDataFrame wrapping the DataFusion DataFrame.

`FlightSQLServer` ¶

A Flight SQL endpoint over lazily registered xarray Datasets.

Register Datasets with register (or xarray_sql.register, which dispatches here), then start it with serve. Datasets registered while the server runs are visible to the next query.

The server runs on a background thread; the Python process must stay alive while it serves. In a script, block with wait. It is also a context manager that shuts down on exit.

`is_running` `property` ¶

Whether the server is accepting connections.

`port` `property` ¶

The TCP port the server is bound to.

`uri` `property` ¶

A grpc:// URI a local client can connect to.

`register(name, ds, *, chunks=None, table_names=None)` ¶

Register ds as a table named name.

A Dataset whose variables sit on different dimensions is split into one table per dimension group, in a SQL schema named name (name.group), exactly as Context.from_dataset registers it.

Names are registered exactly as given. The server's SQL folds unquoted names to lowercase, so a mixed-case name warns and must be quoted in queries.

Parameters:

Name	Type	Description	Default
`name`	`str`	The table name, or the schema name for a mixed-dimension Dataset.	required
`ds`	`Dataset`	An xarray Dataset.	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes.	`None`

Returns:

Type	Description
`FlightSQLServer`	The server, to allow chaining.

`serve(host='127.0.0.1', port=0)` ¶

Start accepting connections on a background thread.

Parameters:

Name	Type	Description	Default
`host`	`str`	The interface to bind. The default accepts only local connections; `"0.0.0.0"` accepts them from anywhere the network allows (the server has no authentication).	`'127.0.0.1'`
`port`	`int`	The TCP port. `0` (default) picks a free one; read it back from port.	`0`

Returns:

Type	Description
`FlightSQLServer`	The server, to allow chaining.

`shutdown(timeout=5.0)` ¶

Stop accepting connections.

Parameters:

Name	Type	Description	Default
`timeout`	`float`	Seconds in-flight queries may keep running, after which their connections are closed. A client that stops reading a result partway otherwise holds its stream open indefinitely.	`5.0`

Raises:

Type	Description
`RuntimeError`	The server had stopped on an error.

`wait(poll_interval=0.5)` ¶

Block until the server stops; Ctrl+C shuts it down.

Raises:

Type	Description
`RuntimeError`	The server stopped on an error.

`getattr(name)` ¶

`arrow_dataset(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, prefetch=DEFAULT_PREFETCH, prefetch_bytes=None, coalesce_rows=None, geometry=None, geometry_encoding='wkb', geometry_crs='OGC:CRS84')` ¶

A pushdown-capable pyarrow.dataset.Dataset view of ds.

The returned object works anywhere a pyarrow dataset does, keeping projection pushdown and coordinate-range chunk pruning::

import polars as pl
lf = pl.scan_pyarrow_dataset(xql.arrow_dataset(ds))

import duckdb
duckdb.connect().register("t", xql.arrow_dataset(ds))

xql.arrow_dataset(ds).to_table(columns=["t2m"], filter=...)

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset. All data variables must share the same dimensions (select a variable subset first otherwise).	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`batch_size`	`int`	Maximum rows per emitted Arrow RecordBatch.	`DEFAULT_BATCH_SIZE`
`prefetch`	`int`	Chunk loads kept in flight ahead of the consumer (memory scales with `prefetch` x pivoted chunk size).	`DEFAULT_PREFETCH`
`prefetch_bytes`	`int \| None`	Optional cap on estimated pivoted bytes in flight; admission then tracks bytes rather than block count, which keeps peak memory steady when `coalesce_rows` makes blocks large or uneven. `prefetch` still bounds concurrency (thread count).	`None`
`coalesce_rows`	`int \| None`	When set, merge runs of consecutive surviving chunks along the most finely chunked dimension into single reads of at most this many rows. Fewer, larger source requests — the win on remote stores, where each merged read fetches its member chunks through the store's own concurrent batching. Memory scales with `prefetch` x the merged block size, so size accordingly (e.g. `8_000_000`).	`None`
`geometry`	`tuple[str, str] \| None`	`(x_dim, y_dim)` coordinate dims to derive a `geometry` point column from (see xarray_sql.geometry). With the default `"wkb"` encoding, DuckDB (spatial loaded) sees a native `GEOMETRY` column, so `ST_Within(geometry, ...)` works directly.	`None`
`geometry_encoding`	`str`	`"wkb"` (default; DuckDB-consumable) or `"point"` (GeoArrow native separated coordinates — the struct children are the coordinate arrays; for GeoPandas, lonboard, geoarrow-rs consumers).	`'wkb'`
`geometry_crs`	`str \| None`	CRS tag carried in the extension metadata. Defaults to `OGC:CRS84` (plain longitude/latitude); pass `None` to omit, or an authority code / PROJJSON string.	`'OGC:CRS84'`

Returns:

Type	Description
`XarrayPushdownDataset`	An XarrayPushdownDataset.

`arrow_datasets(ds, name=None, *, chunks=None, table_names=None, **kwargs)` ¶

Named pushdown datasets, one per dimension group of ds.

arrow_dataset needs a Dataset whose data variables all share one set of dimensions. This applies the same split the DataFusion and DuckDB adapters do — one table per dimension group — and hands back the tables named, ready to register on an engine that has no connection object to dispatch on::

tables = xql.arrow_datasets(ds, 'era5', table_names={
    ('time', 'latitude', 'longitude'): 'surface',
    ('time', 'level', 'latitude', 'longitude'): 'atmosphere',
})

ctx = pl.SQLContext()
for table, dataset in tables.items():   # 'era5_surface', ...
    ctx.register(table, pl.scan_pyarrow_dataset(dataset))

ctx.execute('SELECT AVG("2m_temperature") FROM era5_surface')

Dimension coordinates are read once and shared across the returned tables, which is a network round-trip saved per dimension per group on Zarr-backed stores.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset, with variables on any mix of dimensions.	required
`name`	`str \| None`	Prefix for the table names, as in `era5_surface`. Omit it to get the bare group names (`surface`).	`None`
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Keys naming dimensions a group does not have are ignored for that group. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes, e.g. `{('time', 'latitude', 'longitude'): 'surface'}`. Groups left unnamed take their dimensions joined by underscores (`time_latitude_longitude`); the group holding scalar variables, if any, takes `scalar`.	`None`
`**kwargs`	`Any`	Forwarded to XarrayPushdownDataset (`batch_size`, `prefetch`, ...), applied to every returned table. `geometry=(x_dim, y_dim)` only reaches the groups whose dimensions include both — a mixed-dimension Dataset otherwise has groups the geometry dims are not columns of; those groups come back without a geometry column rather than raising.	`{}`

Returns:

Type	Description
`dict[str, XarrayPushdownDataset]`	Table name to
`dict[str, XarrayPushdownDataset]`	XarrayPushdownDataset.
`dict[str, XarrayPushdownDataset]`	A Dataset with a single dimension group yields one entry, keyed
`dict[str, XarrayPushdownDataset]`	by `name` when given.

`bbox_conjuncts(bounds, x='x', y='y', pad=0.0)` ¶

SQL bbox conjuncts for a geometry's envelope — the pruning half.

Engines do not push ST_* functions into the scan, so a geometry-only predicate reads every chunk; pairing it with range conjuncts on the coordinate columns restores pruning. This helper renders those conjuncts from a geometry's envelope::

poly = shapely.from_wkt("POLYGON (...)")
sql = (
    f"SELECT avg(risk) FROM eri "
    f"WHERE {xql.bbox_conjuncts(poly, x='x', y='y')} "
    f"AND ST_Within(geometry, ST_GeomFromText('{poly.wkt}'))"
)

Parameters:

Name	Type	Description	Default
`bounds`	`Any`	`(xmin, ymin, xmax, ymax)`, or any object with a `.bounds` attribute in that convention (shapely geometries qualify).	required
`x`	`str`	The x/longitude column name.	`'x'`
`y`	`str`	The y/latitude column name.	`'y'`
`pad`	`float`	Optional margin added on every side (e.g. to be safe around `ST_DWithin`-style predicates).	`0.0`

Returns:

Type	Description
`str`	A SQL snippet `"x" BETWEEN a AND b AND "y" BETWEEN c AND d`.

`from_map(func, *iterables, args=None, **kwargs)` ¶

Create a PyArrow Table by mapping a function over iterables.

This is equivalent to dask's from_map but returns a PyArrow Table that can be used with DataFusion instead of a Dask DataFrame.

Parameters:

Name	Type	Description	Default
`func`	`Callable`	Function to apply to each element of the iterables.	required
`*iterables`	`tuple[Any, ...]`	Iterable objects to map the function over.	`()`
`args`	`tuple \| None`	Additional positional arguments to pass to func.	`None`
`**kwargs`	`dict[str, Any]`	Additional keyword arguments to pass to func.	`{}`

Returns:

Type	Description
`Table`	A PyArrow Table containing the concatenated results.

`read_xarray(ds, chunks=None)` ¶

Pivots an Xarray Dataset into a PyArrow Table, partitioned by chunks.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An Xarray Dataset. All `data_vars` must share the same dimensions.	required
`chunks`	`Chunks`	Xarray-like chunks. If not provided, will default to the Dataset's chunks, or for an unchunked Dataset its storage chunks. The product of the chunk sizes becomes the standard length of each dataframe partition.	`None`

Returns:

Type	Description
`RecordBatchReader`	A PyArrow RecordBatchReader, which is a table representation of the input
`RecordBatchReader`	Dataset.

`read_xarray_table(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, coord_arrays=None, _iteration_callback=None)` ¶

Create a lazy DataFusion table from an xarray Dataset.

This is the simplest way to register xarray data with DataFusion. Data is only read when queries are executed, not during registration. The table can be queried multiple times.

Each chunk becomes a separate partition, enabling DataFusion's parallel execution across multiple cores.

Note

SQL queries with WHERE clauses on dimension columns (time, lat, lon, etc.) automatically prune partitions that can't contain matching rows — this is called filter pushdown. For example:

# This query will skip loading partitions with time < '2020-02-01'
result = ctx.sql('SELECT * FROM air WHERE time > "2020-02-01"').collect()

Supported operators: =, <, >, <=, >=, BETWEEN, IN, AND, OR.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset. All data_vars must share the same dimensions.	required
`chunks`	`Chunks`	Xarray-like chunks specification. If not provided, uses the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`batch_size`	`int`	Maximum rows per Arrow RecordBatch emitted per partition. Smaller values let DataFusion start processing earlier; the default (65 536) works well for most datasets.	`DEFAULT_BATCH_SIZE`
`coord_arrays`	`dict[str, ndarray] \| None`	Pre-materialised coordinate arrays keyed by dim-name string. Hand in to share a single read across multiple tables built from the same parent Dataset (e.g. surface + atmosphere from ARCO-ERA5); the dim coords are otherwise read once per `read_xarray_table` call, which is a network round-trip for Zarr-backed datasets.	`None`
`_iteration_callback`	`Callable[[Block, list[str] \| None], None] \| None`	Internal callback for testing. Called with each block dict just before it's converted to Arrow.	`None`

Returns:

Type	Description
`'LazyArrowStreamTable'`	A LazyArrowStreamTable ready for registration with DataFusion.

Example

from datafusion import SessionContext import xarray as xr from xarray_sql import read_xarray_table

ds = xr.tutorial.open_dataset('air_temperature') table = read_xarray_table(ds, chunks={'time': 240})

ctx = SessionContext() ctx.register_table('air', table)

Data is only read here, during query execution¶

Filters on 'time' will prune partitions automatically!¶

result = ctx.sql('SELECT AVG(air) FROM air').collect()

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` ¶

Register a lazy xarray Dataset as a table on an engine connection.

The engine is inferred from the connection type. Data is not read at registration time; the engine pulls Arrow record batches lazily during query execution. Write your SQL in the engine's own dialect and use the engine's extension ecosystem directly — xarray-sql translates the data, not the queries.

Example (DuckDB)::

import duckdb
import xarray_sql as xql

con = duckdb.connect()
xql.register(con, "era5", ds)
rel = con.sql("SELECT time, AVG(t2m) AS t2m FROM era5 GROUP BY time")
result = xql.to_dataset(rel, template=ds)

A Dataset whose variables sit on different dimensions is split into one table per dimension group. Name those tables with table_names, and the same SQL runs on every engine::

xql.register(con, "era5", ds, table_names={
    ("time", "latitude", "longitude"): "surface",
    ("time", "level", "latitude", "longitude"): "atmosphere",
})
con.sql("SELECT AVG(temperature) FROM era5.atmosphere")

Parameters:

Name	Type	Description	Default
`con`	`ConT`	An engine connection: a `datafusion.SessionContext` (or xarray_sql.Context), a `duckdb.DuckDBPyConnection`, or an ADBC DBAPI connection (`adbc_driver_manager.dbapi.Connection`), into which the Dataset is ingested as a table.	required
`name`	`str`	The table name to register the Dataset under. Datasets whose variables have differing dimensions are split into one table per dimension group, addressed as `name.group` on every engine (DuckDB also keeps the flat `name_group` spelling, since its registration namespace is flat).	required
`ds`	`Dataset`	An xarray Dataset.	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks. An unchunked Dataset opened lazily with `chunks=None` is split along the storage chunks its backend recorded in `encoding["preferred_chunks"]`, and is one chunk along other dimensions; one built in memory, which records none, is split into partitions of about a million rows. Dask is not required.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes. Groups left unnamed take their dimensions joined by underscores (`time_latitude_longitude`); the group holding scalar variables, if any, takes `scalar`. Keys matching no group in `ds` are ignored, so one naming map can be reused across Datasets holding different subsets of the same variables.	`None`
`**kwargs`	`Any`	Adapter-specific options, forwarded as-is — e.g. `batch_size` / `prefetch` on DuckDB.	`{}`

Returns:

Type	Description
`ConT`	The connection, to allow chaining.

`serve(datasets, host='127.0.0.1', port=0, *, chunks=None, table_names=None, memory_limit=None)` ¶

Serve Datasets over Arrow Flight SQL, without copying them.

Starts a FlightSQLServer on a background thread with each Dataset registered under its key::

server = xql.serve({"era5": ds}, port=8815)
print(server.uri)   # grpc://127.0.0.1:8815
server.wait()       # in a script: block until Ctrl+C

Any Flight SQL client can then query the Datasets; with ADBC, xql.to_dataset(cursor, template=ds) turns a result back into a labeled Dataset on the client.

Parameters:

Name	Type	Description	Default
`datasets`	`dict[str, Dataset]`	Table name to Dataset.	required
`host`	`str`	The interface to bind; `127.0.0.1` by default.	`'127.0.0.1'`
`port`	`int`	The TCP port; `0` (default) picks a free one.	`0`
`chunks`	`Chunks`	Chunks specification applied to every Dataset.	`None`
`table_names`	`TableNames`	Dimension-group naming applied to every Dataset.	`None`
`memory_limit`	`int \| None`	Bytes queries may hold at once; see FlightSQLServer.	`None`

Returns:

Type	Description
`FlightSQLServer`	The running server.

`to_dataset(result, dims=None, template=None, sparsity='result', fill_value=np.nan, chunks=None, coords='discover', max_result_bytes=None, spill=False)` ¶

Convert an engine's Arrow result into a labeled xr.Dataset.

The engine-agnostic counterpart of XarrayDataFrame.to_dataset: SQL in, array out, for engines xarray-sql does not wrap in a session of its own.

Example (DuckDB)::

con = duckdb.connect()
xql.register(con, "era5", ds)
rel = con.sql(
    "SELECT time, lat, lon, AVG(t2m) AS t2m FROM era5 "
    "GROUP BY time, lat, lon"
)
out = xql.to_dataset(rel, template=ds)

Parameters:

Name	Type	Description	Default
`result`	`Any`	The engine's query result: a `pyarrow.Table`, `RecordBatch` or `RecordBatchReader`, any object implementing `__arrow_c_stream__` (DuckDB relations), or an object with `fetch_record_batch()` / `to_arrow_table()`. The result is consumed once.	required
`dims`	`list[str] \| None`	Result columns to use as Dataset dimensions. When `None`, defaults to the `template`'s dimensions that survive into the result columns (so aggregations that drop dims round-trip on the remaining ones). Either `dims` or `template` must be given.	`None`
`template`	`Dataset \| None`	The source Dataset registered with the engine. Recovers metadata the tabular pivot strips (attrs, encoding, non-dim coordinates, dim-coord dtype) and provides the `dims` default.	`None`
`sparsity`	`Sparsity`	`"result"` (default) keeps only dim values present in the result. `"template"` reindexes to the template's full coord ranges, filling absent cells with `fill_value`.	`'result'`
`fill_value`	`Any`	Fill for `sparsity="template"`. Defaults to NaN.	`nan`
`chunks`	`Mapping[str, int] \| str \| None`	`None` (default) is lazy without a chunk manager, as with `xr.open_dataset(chunks=None)`: each access re-executes the query narrowed to the selection and projected to the variable read, so `.load()` executes it once per variable. Dask is not required, and DuckDB relations work, since every access runs on the calling thread. Because reads re-execute, the connection must stay open while the result is used, later changes to the source show through, and a query whose rows vary between runs (`LIMIT` without `ORDER BY`, random sampling) should be read once with `max_result_bytes=` instead. One-shot streams (Arrow tables, readers, ADBC cursors) have nothing to re-execute and are read eagerly. A mapping (e.g. `{"time": 100}`), `"auto"`, or `"inherit"` selects the lazy/chunked path: data variables are reconstructed window by window on access, each window re-executing the engine's query narrowed to its coordinate range (over a table registered through xarray-sql, that filter flows back into chunk pruning at the source). Requires a re-executable `result` — a Polars LazyFrame/DataFrame or a DataFusion DataFrame; DuckDB relations refuse the chunked path (add `spill=True`).	`None`
`coords`	`Literal['discover', 'template']`	How the lazy path learns each dimension's coordinate values. `"discover"` (default) runs one `DISTINCT` query per dim — correct for any query. `"template"` trusts the template's coord arrays instead, skipping discovery; only valid when the result spans the template's full extent (an unfiltered scan), and requires `template=`.	`'discover'`
`max_result_bytes`	`int \| None`	Optional budget for materializing the result. Setting it reads the result eagerly, in memory. Raises a clean `ValueError` (with the running size) as soon as the materializing result exceeds it — both while collecting the Arrow stream and before allocating the dense arrays — instead of exhausting memory. `None` (default) means unlimited. Results whose only surface is `to_arrow_table()` necessarily materialize in full before the budget can be checked (the check then runs on the materialized size); re-executable results stream instead, so the budget fires before full materialization.	`None`
`spill`	`bool \| str \| PathLike`	Chunked reconstruction from a one-pass on-disk spill instead of per-window re-execution: the result is streamed once (bounded memory) into a temporary Parquet file, and windows re-execute against that file. This serves the two results the re-execution path cannot — DuckDB relations and one-shot Arrow streams — and trades per-window narrowness for a single full pass plus temporary disk. `True` spills to the system temp dir; a path spills into that directory. The file is removed when the returned Dataset is garbage collected. Requires Polars; only valid with `chunks=`.	`False`

Returns: An xr.Dataset with dims as dimensions and the remaining result columns as data variables: lazily indexed by default (in memory for one-shot streams or with max_result_bytes), and lazily chunked when chunks is given.

Raises:

Type	Description
`ValueError`	When neither `dims` nor `template` resolves the dimension columns, a requested dim is missing from the result, or `sparsity="template"` is used without a template.
`TypeError`	When `result` exposes no readable Arrow stream, or `chunks` is requested for a one-shot stream that cannot be re-executed.

`backends` ¶

Engine adapters — the register seam of xarray-sql.

xarray-sql translates data, not queries, across two seams — the two boundaries between xarray and a query engine that neither side builds for itself: register (a lazy xarray.Dataset becomes a table on the engine's own connection; this package) and round-trip (an Arrow result becomes a labeled Dataset again; xarray_sql.to_dataset). SQL dialects, geometry, H3, and optimizers belong to each engine and its extension ecosystem.

Adapters register themselves on import via register_adapter; register dispatches on the connection type.

`EngineAdapter` ¶

Bases: Protocol[ConT]

One engine's implementation of the register seam.

`matches(con)` `staticmethod` ¶

Whether con is a connection this adapter can register into.

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

Register ds as table name on con; returns con.

`FlightSQLServer` ¶

A Flight SQL endpoint over lazily registered xarray Datasets.

Register Datasets with register (or xarray_sql.register, which dispatches here), then start it with serve. Datasets registered while the server runs are visible to the next query.

The server runs on a background thread; the Python process must stay alive while it serves. In a script, block with wait. It is also a context manager that shuts down on exit.

`is_running` `property` ¶

Whether the server is accepting connections.

`port` `property` ¶

The TCP port the server is bound to.

`uri` `property` ¶

A grpc:// URI a local client can connect to.

`register(name, ds, *, chunks=None, table_names=None)` ¶

Register ds as a table named name.

A Dataset whose variables sit on different dimensions is split into one table per dimension group, in a SQL schema named name (name.group), exactly as Context.from_dataset registers it.

Names are registered exactly as given. The server's SQL folds unquoted names to lowercase, so a mixed-case name warns and must be quoted in queries.

Parameters:

Name	Type	Description	Default
`name`	`str`	The table name, or the schema name for a mixed-dimension Dataset.	required
`ds`	`Dataset`	An xarray Dataset.	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes.	`None`

Returns:

Type	Description
`FlightSQLServer`	The server, to allow chaining.

`serve(host='127.0.0.1', port=0)` ¶

Start accepting connections on a background thread.

Parameters:

Name	Type	Description	Default
`host`	`str`	The interface to bind. The default accepts only local connections; `"0.0.0.0"` accepts them from anywhere the network allows (the server has no authentication).	`'127.0.0.1'`
`port`	`int`	The TCP port. `0` (default) picks a free one; read it back from port.	`0`

Returns:

Type	Description
`FlightSQLServer`	The server, to allow chaining.

`shutdown(timeout=5.0)` ¶

Stop accepting connections.

Parameters:

Name	Type	Description	Default
`timeout`	`float`	Seconds in-flight queries may keep running, after which their connections are closed. A client that stops reading a result partway otherwise holds its stream open indefinitely.	`5.0`

Raises:

Type	Description
`RuntimeError`	The server had stopped on an error.

`wait(poll_interval=0.5)` ¶

Block until the server stops; Ctrl+C shuts it down.

Raises:

Type	Description
`RuntimeError`	The server stopped on an error.

`XarrayArrowStream` ¶

A re-scannable Arrow C-stream view over a lazy xarray Dataset.

Arrow PyCapsule consumers (DuckDB among them) call __arrow_c_stream__ once per scan. Each call constructs a fresh XarrayRecordBatchReader over the same lazy Dataset, so — unlike registering a pyarrow.RecordBatchReader directly, which is exhausted after one query — the same registered table supports any number of queries, and data is only read while a query is executing.

The PyCapsule scan path gets no source-level pushdown (the producer never sees the query's columns or filters), so XarrayPushdownDataset is the default registration object; this class remains as the dependency-light fallback.

`XarrayPushdownDataset` ¶

Bases: Dataset

A pushdown-capable pyarrow.dataset.Dataset view of a Dataset.

Consumers that speak the pyarrow dataset protocol (DuckDB, Polars, ...) call scanner with the columns a query needs and the predicate it pushed down; the scan then loads only the needed data variables from only the chunks whose coordinate ranges can satisfy the predicate.

The base class is never initialized (there is no C++ dataset behind this object — the same construction Lance uses for LanceDataset); every entry point consumers touch is overridden in Python, and the few inherited members that would read uninitialized native state are stubbed out.

References

Lance's LanceDataset, a pyarrow.dataset.Dataset subclass built the same way: https://github.com/lancedb/lance (python/python/lance/dataset.py).

`partition_expression` `property` ¶

`schema` `property` ¶

`count_rows(filter=None, **kwargs)` ¶

Count rows, reading as little data as possible.

Without a filter the count is pure chunk arithmetic — no I/O at all. With a filter, chunks are split three ways: pruned chunks contribute nothing, chunks whose coordinate ranges prove the filter true contribute their exact size arithmetically, and only the undecided boundary chunks are scanned (reading just the columns the filter references).

`filter(expression)` ¶

`get_fragments(filter=None)` ¶

One fragment per chunk of the source grid, pruned by filter.

This is how DataFusion consumes the dataset (SessionContext.register_dataset plans one partition per fragment and scans them in parallel), and enables the Dask pattern from_map(lambda f: f.to_table().to_pandas(), ds.get_fragments()).

`join(*args, **kwargs)` ¶

`join_asof(*args, **kwargs)` ¶

`replace_schema(schema)` ¶

`scanner(columns=None, filter=None, batch_size=None, **kwargs)` ¶

Build a scanner for the requested columns and predicate.

filter is applied exactly by the returned scanner (DuckDB deletes the conjuncts it pushes down and trusts the source to enforce them); chunk pruning and column selection only reduce how much data is read to get there. batch_size caps rows per emitted batch (Polars passes it through to_batches). Extra keyword arguments from other pyarrow-dataset consumers are accepted and ignored.

`sort_by(sorting, **kwargs)` ¶

`arrow_dataset(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, prefetch=DEFAULT_PREFETCH, prefetch_bytes=None, coalesce_rows=None, geometry=None, geometry_encoding='wkb', geometry_crs='OGC:CRS84')` ¶

A pushdown-capable pyarrow.dataset.Dataset view of ds.

The returned object works anywhere a pyarrow dataset does, keeping projection pushdown and coordinate-range chunk pruning::

import polars as pl
lf = pl.scan_pyarrow_dataset(xql.arrow_dataset(ds))

import duckdb
duckdb.connect().register("t", xql.arrow_dataset(ds))

xql.arrow_dataset(ds).to_table(columns=["t2m"], filter=...)

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset. All data variables must share the same dimensions (select a variable subset first otherwise).	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`batch_size`	`int`	Maximum rows per emitted Arrow RecordBatch.	`DEFAULT_BATCH_SIZE`
`prefetch`	`int`	Chunk loads kept in flight ahead of the consumer (memory scales with `prefetch` x pivoted chunk size).	`DEFAULT_PREFETCH`
`prefetch_bytes`	`int \| None`	Optional cap on estimated pivoted bytes in flight; admission then tracks bytes rather than block count, which keeps peak memory steady when `coalesce_rows` makes blocks large or uneven. `prefetch` still bounds concurrency (thread count).	`None`
`coalesce_rows`	`int \| None`	When set, merge runs of consecutive surviving chunks along the most finely chunked dimension into single reads of at most this many rows. Fewer, larger source requests — the win on remote stores, where each merged read fetches its member chunks through the store's own concurrent batching. Memory scales with `prefetch` x the merged block size, so size accordingly (e.g. `8_000_000`).	`None`
`geometry`	`tuple[str, str] \| None`	`(x_dim, y_dim)` coordinate dims to derive a `geometry` point column from (see xarray_sql.geometry). With the default `"wkb"` encoding, DuckDB (spatial loaded) sees a native `GEOMETRY` column, so `ST_Within(geometry, ...)` works directly.	`None`
`geometry_encoding`	`str`	`"wkb"` (default; DuckDB-consumable) or `"point"` (GeoArrow native separated coordinates — the struct children are the coordinate arrays; for GeoPandas, lonboard, geoarrow-rs consumers).	`'wkb'`
`geometry_crs`	`str \| None`	CRS tag carried in the extension metadata. Defaults to `OGC:CRS84` (plain longitude/latitude); pass `None` to omit, or an authority code / PROJJSON string.	`'OGC:CRS84'`

Returns:

Type	Description
`XarrayPushdownDataset`	An XarrayPushdownDataset.

`arrow_datasets(ds, name=None, *, chunks=None, table_names=None, **kwargs)` ¶

Named pushdown datasets, one per dimension group of ds.

arrow_dataset needs a Dataset whose data variables all share one set of dimensions. This applies the same split the DataFusion and DuckDB adapters do — one table per dimension group — and hands back the tables named, ready to register on an engine that has no connection object to dispatch on::

tables = xql.arrow_datasets(ds, 'era5', table_names={
    ('time', 'latitude', 'longitude'): 'surface',
    ('time', 'level', 'latitude', 'longitude'): 'atmosphere',
})

ctx = pl.SQLContext()
for table, dataset in tables.items():   # 'era5_surface', ...
    ctx.register(table, pl.scan_pyarrow_dataset(dataset))

ctx.execute('SELECT AVG("2m_temperature") FROM era5_surface')

Dimension coordinates are read once and shared across the returned tables, which is a network round-trip saved per dimension per group on Zarr-backed stores.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset, with variables on any mix of dimensions.	required
`name`	`str \| None`	Prefix for the table names, as in `era5_surface`. Omit it to get the bare group names (`surface`).	`None`
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Keys naming dimensions a group does not have are ignored for that group. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes, e.g. `{('time', 'latitude', 'longitude'): 'surface'}`. Groups left unnamed take their dimensions joined by underscores (`time_latitude_longitude`); the group holding scalar variables, if any, takes `scalar`.	`None`
`**kwargs`	`Any`	Forwarded to XarrayPushdownDataset (`batch_size`, `prefetch`, ...), applied to every returned table. `geometry=(x_dim, y_dim)` only reaches the groups whose dimensions include both — a mixed-dimension Dataset otherwise has groups the geometry dims are not columns of; those groups come back without a geometry column rather than raising.	`{}`

Returns:

Type	Description
`dict[str, XarrayPushdownDataset]`	Table name to
`dict[str, XarrayPushdownDataset]`	XarrayPushdownDataset.
`dict[str, XarrayPushdownDataset]`	A Dataset with a single dimension group yields one entry, keyed
`dict[str, XarrayPushdownDataset]`	by `name` when given.

`get_adapter(con)` ¶

Return the first adapter whose matches(con) is true.

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` ¶

Register a lazy xarray Dataset as a table on an engine connection.

The engine is inferred from the connection type. Data is not read at registration time; the engine pulls Arrow record batches lazily during query execution. Write your SQL in the engine's own dialect and use the engine's extension ecosystem directly — xarray-sql translates the data, not the queries.

Example (DuckDB)::

import duckdb
import xarray_sql as xql

con = duckdb.connect()
xql.register(con, "era5", ds)
rel = con.sql("SELECT time, AVG(t2m) AS t2m FROM era5 GROUP BY time")
result = xql.to_dataset(rel, template=ds)

A Dataset whose variables sit on different dimensions is split into one table per dimension group. Name those tables with table_names, and the same SQL runs on every engine::

xql.register(con, "era5", ds, table_names={
    ("time", "latitude", "longitude"): "surface",
    ("time", "level", "latitude", "longitude"): "atmosphere",
})
con.sql("SELECT AVG(temperature) FROM era5.atmosphere")

Parameters:

Name	Type	Description	Default
`con`	`ConT`	An engine connection: a `datafusion.SessionContext` (or xarray_sql.Context), a `duckdb.DuckDBPyConnection`, or an ADBC DBAPI connection (`adbc_driver_manager.dbapi.Connection`), into which the Dataset is ingested as a table.	required
`name`	`str`	The table name to register the Dataset under. Datasets whose variables have differing dimensions are split into one table per dimension group, addressed as `name.group` on every engine (DuckDB also keeps the flat `name_group` spelling, since its registration namespace is flat).	required
`ds`	`Dataset`	An xarray Dataset.	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks. An unchunked Dataset opened lazily with `chunks=None` is split along the storage chunks its backend recorded in `encoding["preferred_chunks"]`, and is one chunk along other dimensions; one built in memory, which records none, is split into partitions of about a million rows. Dask is not required.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes. Groups left unnamed take their dimensions joined by underscores (`time_latitude_longitude`); the group holding scalar variables, if any, takes `scalar`. Keys matching no group in `ds` are ignored, so one naming map can be reused across Datasets holding different subsets of the same variables.	`None`
`**kwargs`	`Any`	Adapter-specific options, forwarded as-is — e.g. `batch_size` / `prefetch` on DuckDB.	`{}`

Returns:

Type	Description
`ConT`	The connection, to allow chaining.

`register_adapter(cls)` ¶

Class decorator adding an adapter to the dispatch list.

`serve(datasets, host='127.0.0.1', port=0, *, chunks=None, table_names=None, memory_limit=None)` ¶

Serve Datasets over Arrow Flight SQL, without copying them.

Starts a FlightSQLServer on a background thread with each Dataset registered under its key::

server = xql.serve({"era5": ds}, port=8815)
print(server.uri)   # grpc://127.0.0.1:8815
server.wait()       # in a script: block until Ctrl+C

Any Flight SQL client can then query the Datasets; with ADBC, xql.to_dataset(cursor, template=ds) turns a result back into a labeled Dataset on the client.

Parameters:

Name	Type	Description	Default
`datasets`	`dict[str, Dataset]`	Table name to Dataset.	required
`host`	`str`	The interface to bind; `127.0.0.1` by default.	`'127.0.0.1'`
`port`	`int`	The TCP port; `0` (default) picks a free one.	`0`
`chunks`	`Chunks`	Chunks specification applied to every Dataset.	`None`
`table_names`	`TableNames`	Dimension-group naming applied to every Dataset.	`None`
`memory_limit`	`int \| None`	Bytes queries may hold at once; see FlightSQLServer.	`None`

Returns:

Type	Description
`FlightSQLServer`	The running server.

`adbc` ¶

ADBC engine adapter.

ADBC (Arrow Database Connectivity) is a database-neutral API whose drivers speak Arrow natively: PostgreSQL, SQLite, Snowflake, BigQuery, Flight SQL, DuckDB, and more, each behind the same DBAPI-style Connection. This adapter registers a Dataset on any of them through ADBC's bulk-ingest call, so one code path reaches every database with an ADBC driver.

Unlike the DataFusion and DuckDB adapters, registration here copies the data: an ADBC database is typically a separate process or service that cannot call back into Python to scan a lazy Dataset while a query runs. The Dataset is streamed chunk by chunk (prefetched, with bounded memory) into a new table, so only the ingest itself touches the source data, and queries afterwards run entirely inside the database.

Results come back through ADBC's Arrow cursor API, which xarray_sql.to_dataset accepts directly::

cur = con.cursor()
cur.execute("SELECT time, lat, lon, t2m FROM era5 ORDER BY time")
out = xql.to_dataset(cur, template=ds)

This adapter never imports adbc_driver_manager at runtime — detection is by connection type — so ADBC stays a purely optional dependency (pip install xarray-sql[adbc] plus a driver package).

`ADBCAdapter` ¶

Registers Datasets on ADBC DBAPI connections.

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, mode='create', temporary=False, ingest_options=None, **kwargs)` `staticmethod` ¶

Ingest ds into tables on an ADBC connection.

Datasets whose variables all share the same dimensions become a single table named name. Mixed-dimension datasets are split into one table per dimension group, created in a database schema named name so they are queried as name.group — the same spelling every other engine uses::

xql.register(con, 'era5', ds, table_names={
    ('time', 'latitude', 'longitude'): 'surface',
    ('time', 'level', 'latitude', 'longitude'): 'atmosphere',
})
cur.execute('SELECT ... FROM era5.surface')

An existing schema is used as is. On databases without schemas (SQLite), and for temporary tables, which most drivers cannot place in a schema, the groups are created as flat name_group tables instead. If creating the schema fails in a way that aborts the transaction (PostgreSQL without the CREATE privilege), this raises RuntimeError; roll back, then create the schema beforehand or pass temporary=True.

On ClickHouse, whose driver can only append, the adapter creates the tables itself: MergeTree tables sorted by their dimensions (Memory for temporary ones), in a ClickHouse database named name for a mixed-dimension Dataset, with timestamps declared DateTime64(p, 'UTC') at the coordinate's precision (p is 9 for datetime64[ns]).

Registration runs inside the connection's current transaction: the tables are visible to this connection immediately, and to others once you call con.commit() (unless the connection is in autocommit mode).

Parameters:

Name	Type	Description	Default
`mode`	`IngestMode`	What to do when a table already exists, as in ADBC's `adbc_ingest`: `"create"` (default) raises, `"replace"` drops and recreates it, `"append"` and `"create_append"` add rows to it.	`'create'`
`temporary`	`bool`	Create temporary tables, which the database drops when the connection closes. Raises `ValueError` on databases whose driver cannot (DataFusion, Trino, Spark, BigQuery, Databricks, Snowflake).	`False`
`ingest_options`	`Mapping[str, str] \| None`	Driver-specific statement options set on each ingest, e.g. Spark's `{"spark.ingest.staging_area_uri": "s3://bucket/path"}`.	`None`
`**kwargs`	`Any`	Forwarded to XarrayPushdownDataset (`batch_size`, `prefetch`, `prefetch_bytes`, `coalesce_rows`) to tune the ingest scan.	`{}`

`base` ¶

Engine-adapter dispatch for xarray_sql.register.

An engine adapter implements the register seam: given an engine's native connection object and a lazy xarray.Dataset, register the Dataset as a queryable table on that connection. The Arrow C-stream protocol is the common wire between xarray and every engine; adapters differ only in how a stream is attached to the connection and in what pushdown the engine can do against it.

Adapters self-describe which connections they accept via matches, which must not require the engine's package to be importable (detection is by type inspection), so optional engines stay optional.

`ConT = TypeVar('ConT')` `module-attribute` ¶

An engine's native connection type (e.g. duckdb.DuckDBPyConnection).

`EngineAdapter` ¶

Bases: Protocol[ConT]

One engine's implementation of the register seam.

`matches(con)` `staticmethod` ¶

Whether con is a connection this adapter can register into.

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

Register ds as table name on con; returns con.

`get_adapter(con)` ¶

Return the first adapter whose matches(con) is true.

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` ¶

Register a lazy xarray Dataset as a table on an engine connection.

The engine is inferred from the connection type. Data is not read at registration time; the engine pulls Arrow record batches lazily during query execution. Write your SQL in the engine's own dialect and use the engine's extension ecosystem directly — xarray-sql translates the data, not the queries.

Example (DuckDB)::

import duckdb
import xarray_sql as xql

con = duckdb.connect()
xql.register(con, "era5", ds)
rel = con.sql("SELECT time, AVG(t2m) AS t2m FROM era5 GROUP BY time")
result = xql.to_dataset(rel, template=ds)

A Dataset whose variables sit on different dimensions is split into one table per dimension group. Name those tables with table_names, and the same SQL runs on every engine::

xql.register(con, "era5", ds, table_names={
    ("time", "latitude", "longitude"): "surface",
    ("time", "level", "latitude", "longitude"): "atmosphere",
})
con.sql("SELECT AVG(temperature) FROM era5.atmosphere")

Parameters:

Name	Type	Description	Default
`con`	`ConT`	An engine connection: a `datafusion.SessionContext` (or xarray_sql.Context), a `duckdb.DuckDBPyConnection`, or an ADBC DBAPI connection (`adbc_driver_manager.dbapi.Connection`), into which the Dataset is ingested as a table.	required
`name`	`str`	The table name to register the Dataset under. Datasets whose variables have differing dimensions are split into one table per dimension group, addressed as `name.group` on every engine (DuckDB also keeps the flat `name_group` spelling, since its registration namespace is flat).	required
`ds`	`Dataset`	An xarray Dataset.	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks. An unchunked Dataset opened lazily with `chunks=None` is split along the storage chunks its backend recorded in `encoding["preferred_chunks"]`, and is one chunk along other dimensions; one built in memory, which records none, is split into partitions of about a million rows. Dask is not required.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes. Groups left unnamed take their dimensions joined by underscores (`time_latitude_longitude`); the group holding scalar variables, if any, takes `scalar`. Keys matching no group in `ds` are ignored, so one naming map can be reused across Datasets holding different subsets of the same variables.	`None`
`**kwargs`	`Any`	Adapter-specific options, forwarded as-is — e.g. `batch_size` / `prefetch` on DuckDB.	`{}`

Returns:

Type	Description
`ConT`	The connection, to allow chaining.

`register_adapter(cls)` ¶

Class decorator adding an adapter to the dispatch list.

`datafusion` ¶

DataFusion engine adapter.

DataFusion is xarray-sql's default engine and the richest adapter: the Rust LazyArrowStreamTable table provider gives partition pruning on dimension predicates, projection pushdown, and exact per-partition statistics for the optimizer. This module only routes the generic xarray_sql.register seam onto that existing machinery.

Like the other adapters, it never imports datafusion until a DataFusion connection is registered, so the rest of xarray-sql works where DataFusion's Python bindings are unavailable (e.g. Pyodide).

`DataFusionAdapter` ¶

Registers Datasets on datafusion.SessionContext connections.

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

`duckdb` ¶

DuckDB engine adapter.

Registers a lazy xarray.Dataset on a duckdb.DuckDBPyConnection as an XarrayPushdownDataset: DuckDB classifies it with a real isinstance check against pyarrow.dataset.Dataset and calls scanner(columns=[...], filter=<pyarrow.compute.Expression>) once per query, giving projection pushdown, coordinate-range chunk pruning, and prefetched parallel production (see xarray_sql.backends.pyarrow).

This adapter never imports the duckdb package at runtime — detection is by connection type, and registration is a method call on the connection — so DuckDB stays a purely optional dependency (pip install xarray-sql[duckdb]).

Zarr-native scanning inside DuckDB is what the duckdb-zarr extension provides; this adapter instead covers everything xarray can open (NetCDF, GRIB, Xee, CF decoding, in-memory) and pairs with xarray_sql.to_dataset for the labeled round-trip.

`DuckDBAdapter` ¶

Registers Datasets on duckdb.DuckDBPyConnection connections.

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

Register ds on a DuckDB connection.

Datasets whose variables all share the same dimensions become a single table named name. Mixed-dimension datasets are split into one table per dimension group, named after the group's dimensions joined by underscores unless table_names gives the group a name of its own::

xql.register(con, 'era5', ds, table_names={
    ('time', 'latitude', 'longitude'): 'surface',
    ('time', 'level', 'latitude', 'longitude'): 'atmosphere',
})
con.sql('SELECT ... FROM era5.surface')   # or era5_surface

Each group is registered under the flat name <name>_<group> and mirrored as a view <name>.<group> in a DuckDB schema, so the dotted spelling the DataFusion adapter uses queries the same table here. Extra keyword arguments (batch_size, prefetch) are forwarded to XarrayPushdownDataset.

`XarrayArrowStream` ¶

A re-scannable Arrow C-stream view over a lazy xarray Dataset.

Arrow PyCapsule consumers (DuckDB among them) call __arrow_c_stream__ once per scan. Each call constructs a fresh XarrayRecordBatchReader over the same lazy Dataset, so — unlike registering a pyarrow.RecordBatchReader directly, which is exhausted after one query — the same registered table supports any number of queries, and data is only read while a query is executing.

The PyCapsule scan path gets no source-level pushdown (the producer never sees the query's columns or filters), so XarrayPushdownDataset is the default registration object; this class remains as the dependency-light fallback.

`XarrayPushdownDataset` ¶

Bases: Dataset

A pushdown-capable pyarrow.dataset.Dataset view of a Dataset.

Consumers that speak the pyarrow dataset protocol (DuckDB, Polars, ...) call scanner with the columns a query needs and the predicate it pushed down; the scan then loads only the needed data variables from only the chunks whose coordinate ranges can satisfy the predicate.

The base class is never initialized (there is no C++ dataset behind this object — the same construction Lance uses for LanceDataset); every entry point consumers touch is overridden in Python, and the few inherited members that would read uninitialized native state are stubbed out.

References

Lance's LanceDataset, a pyarrow.dataset.Dataset subclass built the same way: https://github.com/lancedb/lance (python/python/lance/dataset.py).

`partition_expression` `property` ¶

`schema` `property` ¶

`count_rows(filter=None, **kwargs)` ¶

Count rows, reading as little data as possible.

Without a filter the count is pure chunk arithmetic — no I/O at all. With a filter, chunks are split three ways: pruned chunks contribute nothing, chunks whose coordinate ranges prove the filter true contribute their exact size arithmetically, and only the undecided boundary chunks are scanned (reading just the columns the filter references).

`filter(expression)` ¶

`get_fragments(filter=None)` ¶

One fragment per chunk of the source grid, pruned by filter.

This is how DataFusion consumes the dataset (SessionContext.register_dataset plans one partition per fragment and scans them in parallel), and enables the Dask pattern from_map(lambda f: f.to_table().to_pandas(), ds.get_fragments()).

`join(*args, **kwargs)` ¶

`join_asof(*args, **kwargs)` ¶

`replace_schema(schema)` ¶

`scanner(columns=None, filter=None, batch_size=None, **kwargs)` ¶

Build a scanner for the requested columns and predicate.

filter is applied exactly by the returned scanner (DuckDB deletes the conjuncts it pushes down and trusts the source to enforce them); chunk pruning and column selection only reduce how much data is read to get there. batch_size caps rows per emitted batch (Polars passes it through to_batches). Extra keyword arguments from other pyarrow-dataset consumers are accepted and ignored.

`sort_by(sorting, **kwargs)` ¶

`flight` ¶

Serve lazy xarray Datasets over Arrow Flight SQL.

The other adapters bring an engine to the data: the Dataset becomes a table on a connection in this process. A FlightSQLServer turns that around and brings the data to remote clients. It hosts a DataFusion session over the same lazy tables Context uses and speaks Flight SQL, so any Flight SQL client — ADBC's Flight SQL driver in Python, R, Go, or Java, the Flight SQL JDBC and ODBC drivers, and the SQL tools built on them — can query the Dataset without a copy::

server = xql.serve({"era5": ds}, port=8815)

# elsewhere
import adbc_driver_flightsql.dbapi as flight_sql
con = flight_sql.connect("grpc://server-host:8815")
cur = con.cursor()
cur.execute("SELECT AVG(t2m) FROM era5 WHERE time >= '2020-01-01'")

Queries keep partition pruning on dimension predicates and projection pushdown: only the chunks and variables a query touches are read from the source, only while it runs, and results stream back as Arrow record batches.

The server has no authentication or TLS, and binds to 127.0.0.1 unless told otherwise. Its SQL is read-only: DDL, DML, and other statements are rejected, so clients cannot read or write the server's filesystem through it.

`FlightSQLAdapter` ¶

Registers Datasets on a FlightSQLServer.

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

`FlightSQLServer` ¶

A Flight SQL endpoint over lazily registered xarray Datasets.

Register Datasets with register (or xarray_sql.register, which dispatches here), then start it with serve. Datasets registered while the server runs are visible to the next query.

The server runs on a background thread; the Python process must stay alive while it serves. In a script, block with wait. It is also a context manager that shuts down on exit.

`is_running` `property` ¶

Whether the server is accepting connections.

`port` `property` ¶

The TCP port the server is bound to.

`uri` `property` ¶

A grpc:// URI a local client can connect to.

`register(name, ds, *, chunks=None, table_names=None)` ¶

Register ds as a table named name.

A Dataset whose variables sit on different dimensions is split into one table per dimension group, in a SQL schema named name (name.group), exactly as Context.from_dataset registers it.

Names are registered exactly as given. The server's SQL folds unquoted names to lowercase, so a mixed-case name warns and must be quoted in queries.

Parameters:

Name	Type	Description	Default
`name`	`str`	The table name, or the schema name for a mixed-dimension Dataset.	required
`ds`	`Dataset`	An xarray Dataset.	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes.	`None`

Returns:

Type	Description
`FlightSQLServer`	The server, to allow chaining.

`serve(host='127.0.0.1', port=0)` ¶

Start accepting connections on a background thread.

Parameters:

Name	Type	Description	Default
`host`	`str`	The interface to bind. The default accepts only local connections; `"0.0.0.0"` accepts them from anywhere the network allows (the server has no authentication).	`'127.0.0.1'`
`port`	`int`	The TCP port. `0` (default) picks a free one; read it back from port.	`0`

Returns:

Type	Description
`FlightSQLServer`	The server, to allow chaining.

`shutdown(timeout=5.0)` ¶

Stop accepting connections.

Parameters:

Name	Type	Description	Default
`timeout`	`float`	Seconds in-flight queries may keep running, after which their connections are closed. A client that stops reading a result partway otherwise holds its stream open indefinitely.	`5.0`

Raises:

Type	Description
`RuntimeError`	The server had stopped on an error.

`wait(poll_interval=0.5)` ¶

Block until the server stops; Ctrl+C shuts it down.

Raises:

Type	Description
`RuntimeError`	The server stopped on an error.

`serve(datasets, host='127.0.0.1', port=0, *, chunks=None, table_names=None, memory_limit=None)` ¶

Serve Datasets over Arrow Flight SQL, without copying them.

Starts a FlightSQLServer on a background thread with each Dataset registered under its key::

server = xql.serve({"era5": ds}, port=8815)
print(server.uri)   # grpc://127.0.0.1:8815
server.wait()       # in a script: block until Ctrl+C

Any Flight SQL client can then query the Datasets; with ADBC, xql.to_dataset(cursor, template=ds) turns a result back into a labeled Dataset on the client.

Parameters:

Name	Type	Description	Default
`datasets`	`dict[str, Dataset]`	Table name to Dataset.	required
`host`	`str`	The interface to bind; `127.0.0.1` by default.	`'127.0.0.1'`
`port`	`int`	The TCP port; `0` (default) picks a free one.	`0`
`chunks`	`Chunks`	Chunks specification applied to every Dataset.	`None`
`table_names`	`TableNames`	Dimension-group naming applied to every Dataset.	`None`
`memory_limit`	`int \| None`	Bytes queries may hold at once; see FlightSQLServer.	`None`

Returns:

Type	Description
`FlightSQLServer`	The running server.

`pyarrow` ¶

Engine-neutral pyarrow views of lazy xarray Datasets.

Two ways to hand a lazy xarray.Dataset to an Arrow-speaking query engine:

XarrayPushdownDataset — a real pyarrow.dataset.Dataset subclass (the pattern Lance uses for LanceDataset). Consumers of the pyarrow dataset protocol — DuckDB via con.register, Polars via pl.scan_pyarrow_dataset, or pyarrow itself — call scanner with the columns a query needs and the predicate it pushed down, so the scan loads only the needed data variables from only the chunks whose coordinate ranges can satisfy the predicate. Construct one with arrow_dataset.
XarrayArrowStream — a re-scannable Arrow C-stream (PyCapsule) view. No source-level pushdown, but works with any PyCapsule consumer; the dependency-light fallback.

Correctness contract shared by all consumers of the pushdown dataset: engines may delete the filter conjuncts they push down (DuckDB does), so the returned scanner applies the expression exactly via pyarrow.dataset.Scanner; chunk pruning is only ever an optimization on top.

`DEFAULT_PREFETCH = 4 if _CAN_START_THREADS else 1` `module-attribute` ¶

Chunk loads kept in flight ahead of the consumer during a scan.

Where threads are unavailable (Pyodide), chunks load one at a time.

`XarrayArrowStream` ¶

A re-scannable Arrow C-stream view over a lazy xarray Dataset.

Arrow PyCapsule consumers (DuckDB among them) call __arrow_c_stream__ once per scan. Each call constructs a fresh XarrayRecordBatchReader over the same lazy Dataset, so — unlike registering a pyarrow.RecordBatchReader directly, which is exhausted after one query — the same registered table supports any number of queries, and data is only read while a query is executing.

The PyCapsule scan path gets no source-level pushdown (the producer never sees the query's columns or filters), so XarrayPushdownDataset is the default registration object; this class remains as the dependency-light fallback.

`XarrayPushdownDataset` ¶

Bases: Dataset

A pushdown-capable pyarrow.dataset.Dataset view of a Dataset.

Consumers that speak the pyarrow dataset protocol (DuckDB, Polars, ...) call scanner with the columns a query needs and the predicate it pushed down; the scan then loads only the needed data variables from only the chunks whose coordinate ranges can satisfy the predicate.

The base class is never initialized (there is no C++ dataset behind this object — the same construction Lance uses for LanceDataset); every entry point consumers touch is overridden in Python, and the few inherited members that would read uninitialized native state are stubbed out.

References

Lance's LanceDataset, a pyarrow.dataset.Dataset subclass built the same way: https://github.com/lancedb/lance (python/python/lance/dataset.py).

`partition_expression` `property` ¶

`schema` `property` ¶

`count_rows(filter=None, **kwargs)` ¶

Count rows, reading as little data as possible.

Without a filter the count is pure chunk arithmetic — no I/O at all. With a filter, chunks are split three ways: pruned chunks contribute nothing, chunks whose coordinate ranges prove the filter true contribute their exact size arithmetically, and only the undecided boundary chunks are scanned (reading just the columns the filter references).

`filter(expression)` ¶

`get_fragments(filter=None)` ¶

One fragment per chunk of the source grid, pruned by filter.

This is how DataFusion consumes the dataset (SessionContext.register_dataset plans one partition per fragment and scans them in parallel), and enables the Dask pattern from_map(lambda f: f.to_table().to_pandas(), ds.get_fragments()).

`join(*args, **kwargs)` ¶

`join_asof(*args, **kwargs)` ¶

`replace_schema(schema)` ¶

`scanner(columns=None, filter=None, batch_size=None, **kwargs)` ¶

Build a scanner for the requested columns and predicate.

filter is applied exactly by the returned scanner (DuckDB deletes the conjuncts it pushes down and trusts the source to enforce them); chunk pruning and column selection only reduce how much data is read to get there. batch_size caps rows per emitted batch (Polars passes it through to_batches). Extra keyword arguments from other pyarrow-dataset consumers are accepted and ignored.

`sort_by(sorting, **kwargs)` ¶

`arrow_dataset(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, prefetch=DEFAULT_PREFETCH, prefetch_bytes=None, coalesce_rows=None, geometry=None, geometry_encoding='wkb', geometry_crs='OGC:CRS84')` ¶

A pushdown-capable pyarrow.dataset.Dataset view of ds.

The returned object works anywhere a pyarrow dataset does, keeping projection pushdown and coordinate-range chunk pruning::

import polars as pl
lf = pl.scan_pyarrow_dataset(xql.arrow_dataset(ds))

import duckdb
duckdb.connect().register("t", xql.arrow_dataset(ds))

xql.arrow_dataset(ds).to_table(columns=["t2m"], filter=...)

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset. All data variables must share the same dimensions (select a variable subset first otherwise).	required
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`batch_size`	`int`	Maximum rows per emitted Arrow RecordBatch.	`DEFAULT_BATCH_SIZE`
`prefetch`	`int`	Chunk loads kept in flight ahead of the consumer (memory scales with `prefetch` x pivoted chunk size).	`DEFAULT_PREFETCH`
`prefetch_bytes`	`int \| None`	Optional cap on estimated pivoted bytes in flight; admission then tracks bytes rather than block count, which keeps peak memory steady when `coalesce_rows` makes blocks large or uneven. `prefetch` still bounds concurrency (thread count).	`None`
`coalesce_rows`	`int \| None`	When set, merge runs of consecutive surviving chunks along the most finely chunked dimension into single reads of at most this many rows. Fewer, larger source requests — the win on remote stores, where each merged read fetches its member chunks through the store's own concurrent batching. Memory scales with `prefetch` x the merged block size, so size accordingly (e.g. `8_000_000`).	`None`
`geometry`	`tuple[str, str] \| None`	`(x_dim, y_dim)` coordinate dims to derive a `geometry` point column from (see xarray_sql.geometry). With the default `"wkb"` encoding, DuckDB (spatial loaded) sees a native `GEOMETRY` column, so `ST_Within(geometry, ...)` works directly.	`None`
`geometry_encoding`	`str`	`"wkb"` (default; DuckDB-consumable) or `"point"` (GeoArrow native separated coordinates — the struct children are the coordinate arrays; for GeoPandas, lonboard, geoarrow-rs consumers).	`'wkb'`
`geometry_crs`	`str \| None`	CRS tag carried in the extension metadata. Defaults to `OGC:CRS84` (plain longitude/latitude); pass `None` to omit, or an authority code / PROJJSON string.	`'OGC:CRS84'`

Returns:

Type	Description
`XarrayPushdownDataset`	An XarrayPushdownDataset.

`arrow_datasets(ds, name=None, *, chunks=None, table_names=None, **kwargs)` ¶

Named pushdown datasets, one per dimension group of ds.

arrow_dataset needs a Dataset whose data variables all share one set of dimensions. This applies the same split the DataFusion and DuckDB adapters do — one table per dimension group — and hands back the tables named, ready to register on an engine that has no connection object to dispatch on::

tables = xql.arrow_datasets(ds, 'era5', table_names={
    ('time', 'latitude', 'longitude'): 'surface',
    ('time', 'level', 'latitude', 'longitude'): 'atmosphere',
})

ctx = pl.SQLContext()
for table, dataset in tables.items():   # 'era5_surface', ...
    ctx.register(table, pl.scan_pyarrow_dataset(dataset))

ctx.execute('SELECT AVG("2m_temperature") FROM era5_surface')

Dimension coordinates are read once and shared across the returned tables, which is a network round-trip saved per dimension per group on Zarr-backed stores.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset, with variables on any mix of dimensions.	required
`name`	`str \| None`	Prefix for the table names, as in `era5_surface`. Omit it to get the bare group names (`surface`).	`None`
`chunks`	`Chunks`	Xarray-like chunks specification controlling partition granularity. Keys naming dimensions a group does not have are ignored for that group. Defaults to the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`table_names`	`TableNames`	Maps a dimension group's exact dim tuple to the name its table takes, e.g. `{('time', 'latitude', 'longitude'): 'surface'}`. Groups left unnamed take their dimensions joined by underscores (`time_latitude_longitude`); the group holding scalar variables, if any, takes `scalar`.	`None`
`**kwargs`	`Any`	Forwarded to XarrayPushdownDataset (`batch_size`, `prefetch`, ...), applied to every returned table. `geometry=(x_dim, y_dim)` only reaches the groups whose dimensions include both — a mixed-dimension Dataset otherwise has groups the geometry dims are not columns of; those groups come back without a geometry column rather than raising.	`{}`

Returns:

Type	Description
`dict[str, XarrayPushdownDataset]`	Table name to
`dict[str, XarrayPushdownDataset]`	XarrayPushdownDataset.
`dict[str, XarrayPushdownDataset]`	A Dataset with a single dimension group yields one entry, keyed
`dict[str, XarrayPushdownDataset]`	by `name` when given.

`cftime` ¶

Bridge between cftime calendars and Arrow/DataFusion types.

cftime (https://unidata.github.io/cftime/) provides datetime objects for calendars used in climate science — noleap, 360-day, all-leap, julian, etc. Arrow and DataFusion have no native concept of non-Gregorian calendars, so this module handles the conversion in two tiers:

Gregorian-like calendars (standard, gregorian, proleptic_gregorian, noleap/365_day, all_leap/366_day): mapped to pa.timestamp('us') so that string-based SQL filters like WHERE time > '1980-01-01' work naturally. Microsecond resolution avoids the 1678–2262 overflow of nanoseconds while preserving sub-second precision.
Non-Gregorian calendars (360_day, julian): mapped to pa.int64() with xarray:units and xarray:calendar metadata on the Arrow field. This preserves the original CF-convention encoding losslessly. A cftime() DataFusion UDF (registered automatically by Context.from_dataset) provides ergonomic SQL filtering.

`DEFAULT_UNITS = 'microseconds since 1970-01-01T00:00:00'` `module-attribute` ¶

Default CF-convention units when no encoding is available on the coordinate.

Microseconds give sub-second precision and fit int64 for ±292 k years.

`GREGORIAN_LIKE_CALENDARS = frozenset({'standard', 'gregorian', 'proleptic_gregorian', 'noleap', '365_day', 'all_leap', '366_day'})` `module-attribute` ¶

Calendars close enough to proleptic Gregorian for pa.timestamp('us').

`arrow_field(name, units, cal)` ¶

Build a pa.Field for a cftime coordinate.

Gregorian-like → pa.timestamp('us'); non-Gregorian → pa.int64(). Both carry xarray:calendar and xarray:units metadata for round-trip fidelity.

`calendar(ds, coord_name)` ¶

Return the calendar name for a cftime coordinate, or None.

Checks the xarray index first (no data materialization), then falls back to inspecting element 0 of the coordinate values.

`convert_for_field(values, field)` ¶

Convert cftime values to the numeric type dictated by field.

Reads xarray:calendar and xarray:units from the field's Arrow metadata to choose between the timestamp path and the integer-offset path.

`encoding(ds, coord_name)` ¶

Return (units, calendar) for a cftime coordinate.

Reads xarray .encoding metadata (from the originating NetCDF file) first, falling back to DEFAULT_UNITS.

`is_cftime(values)` ¶

Check if a numpy array contains cftime datetime objects.

`is_cftime_index(ds, coord_name)` ¶

Check if a coordinate uses a CFTimeIndex without materializing data.

`is_gregorian_like(calendar)` ¶

Return True if calendar is close enough to Gregorian for pa.timestamp.

`make_cftime_udf(units, calendar)` ¶

Create a DataFusion scalar UDF that converts date strings to int64 offsets.

This enables ergonomic SQL filtering on non-Gregorian cftime columns::

SELECT * FROM ds360 WHERE time > cftime('0500-01-01')

The UDF parses the input string as a cftime datetime in the given calendar system and returns the corresponding int64 offset in the specified units.

`partition_bounds(values)` ¶

Return (min, max, dtype_tag) for a cftime coordinate slice.

Gregorian-like calendars return nanosecond bounds tagged "timestamp_ns" (compatible with ScalarBound::TimestampNanos in the Rust pruning layer). Non-Gregorian calendars return int64 offsets tagged "int64".

Returns None when the nanosecond bound falls outside the int64 range (e.g. paleoclimate dates before ~1678), signalling the caller to skip pruning for that dimension rather than emit a bound the Rust layer would reject.

`to_microseconds(values)` ¶

Convert cftime objects to int64 microseconds since Unix epoch.

Used for Gregorian-like calendars. Vectorised via cftime.date2num (implemented in C).

`to_offsets(values, units, cal)` ¶

Convert cftime objects to int64 offsets in the given units/calendar.

Used for non-Gregorian calendars where data is stored as pa.int64().

`core` ¶

`Row = list[Any]` `module-attribute` ¶

`get_columns(ds)` ¶

`unbounded_unravel(ds)` ¶

Unravel with unbounded memory (as a NumPy Array).

`unravel(ds)` ¶

`df` ¶

`Block = dict[Hashable, slice]` `module-attribute` ¶

`Chunks = dict[str, int] | None` `module-attribute` ¶

`DEFAULT_BATCH_SIZE = 65536` `module-attribute` ¶

Default number of rows per emitted Arrow RecordBatch.

64 K rows balances DataFusion pipeline depth against per-batch overhead.

`PartitionBounds = dict[str, tuple[Any, Any, str]]` `module-attribute` ¶

`TableNames = Mapping[tuple[str, ...], str] | None` `module-attribute` ¶

Maps a dimension group's exact dim tuple to the table name it takes.

`block_slices(ds, chunks=None)` ¶

Compute block slices for a chunked Dataset.

`compute_chunks(ds, chunks)` ¶

Per-dim chunk-size tuples matching ds.chunk(chunks).chunks.

Pure arithmetic replacement for the dask rechunk round-trip; dask's .chunk() eagerly builds a task graph, which dominates block_slices() cost on large datasets.

`dataset_to_record_batch(ds, schema)` ¶

Convert an xarray Dataset partition to an Arrow RecordBatch.

Builds the RecordBatch directly from numpy arrays, bypassing the pandas round-trip (to_dataframe → reset_index → from_pandas) used by pivot(). For large partitions this reduces peak memory from ~5× to ~2× the partition size.

Dimension coordinates are broadcast to the full partition shape and ravelled. np.broadcast_to() is zero-copy; the ravel() forces one copy per coordinate (unavoidable, since broadcast arrays are non-contiguous). Data variable arrays are ravelled in-place — a zero-copy view when the underlying array is already C-contiguous (the common case for numpy-backed xarray datasets).

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	A partition-sized xarray Dataset (already sliced via isel).	required
`schema`	`Schema`	The Arrow schema for the output, as produced by _parse_schema. Column order in the output matches schema field order.	required

Returns:

Type	Description
`RecordBatch`	A RecordBatch with one column per dimension coordinate and data
`RecordBatch`	variable, in schema order.

`default_table_name(dims)` ¶

The table name a dimension group gets when the user names none.

`explode(ds, chunks=None)` ¶

Explodes a dataset into its chunks.

`from_map(func, *iterables, args=None, **kwargs)` ¶

Create a PyArrow Table by mapping a function over iterables.

This is equivalent to dask's from_map but returns a PyArrow Table that can be used with DataFusion instead of a Dask DataFrame.

Parameters:

Name	Type	Description	Default
`func`	`Callable`	Function to apply to each element of the iterables.	required
`*iterables`	`tuple[Any, ...]`	Iterable objects to map the function over.	`()`
`args`	`tuple \| None`	Additional positional arguments to pass to func.	`None`
`**kwargs`	`dict[str, Any]`	Additional keyword arguments to pass to func.	`{}`

Returns:

Type	Description
`Table`	A PyArrow Table containing the concatenated results.

`from_map_batched(func, *iterables, args=None, schema=None, **kwargs)` ¶

Create a PyArrow RecordBatchReader by mapping a function over iterables.

This is equivalent to dask's from_map but returns a PyArrow RecordBatchReader that can be used with DataFusion. It iterates over RecordBatches which are created via the func one-at-a-time.

Parameters:

Name	Type	Description	Default
`func`	`Callable[..., DataFrame]`	Function to apply to each element of the iterables. Currently, the function must return a Pandas DataFrame.	required
`*iterables`	`tuple[Any, ...]`	Iterable objects to map the function over.	`()`
`schema`	`Schema`	Optional schema needed for the RecordBatchReader.	`None`
`args`	`tuple \| None`	Additional positional arguments to pass to func.	`None`
`**kwargs`	`dict[str, Any]`	Additional keyword arguments to pass to func.	`{}`

Returns:

Type	Description
`RecordBatchReader`	A PyArrow RecordBatchReader containing the stream of RecordBatches.

`group_vars_by_dims(ds)` ¶

Group a Dataset's data variables by their exact dimension tuple.

Variables that share dimensions can share a table; each distinct dimension tuple becomes its own table when a mixed-dimension Dataset is registered::

("time", "lat", "lon"):          ["temperature_2m", "wind_speed"],
("time", "lat", "lon", "level"): ["pressure", "humidity"]

`iter_record_batches(ds, schema, batch_size=DEFAULT_BATCH_SIZE)` ¶

Yield RecordBatches of at most batch_size rows from a partition Dataset.

Unlike dataset_to_record_batch, which materialises the entire partition as one batch, this generator emits smaller batches so that DataFusion can begin filtering and aggregating before the full partition is loaded. Peak memory per batch is O(batch_size) for coordinate columns and O(partition_size) for data-variable columns (which must be loaded in full from storage).

Coordinate values are computed per batch via strided index arithmetic — no broadcast array spanning the whole partition is ever allocated. Data variable flat arrays are loaded once (triggering any remote I/O) and then sliced as zero-copy views for each batch.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	A partition-sized xarray Dataset (already sliced via isel).	required
`schema`	`Schema`	The Arrow schema for the output, as produced by _parse_schema.	required
`batch_size`	`int`	Maximum number of rows per yielded RecordBatch.	`DEFAULT_BATCH_SIZE`

Yields:

Type	Description
`RecordBatch`	RecordBatches in schema column order, covering all rows of the
`RecordBatch`	partition exactly once.

`partition_metadata(ds, blocks)` ¶

Compute min/max coordinate values for each partition.

This metadata enables filter pushdown: SQL queries with WHERE clauses on dimension columns can prune partitions that can't contain matching rows.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	The xarray Dataset containing coordinate values.	required
`blocks`	`list[Block]`	List of block slices from block_slices().	required

Returns:

Type	Description
`list[PartitionBounds]`	List of dicts mapping dimension name to
`list[PartitionBounds]`	(min_value, max_value, dtype_str) tuples.
`list[PartitionBounds]`	Values preserve the coordinate's logical type and width. Temporal
`list[PartitionBounds]`	values are signed integer counts in their Arrow unit; strings and
`list[PartitionBounds]`	bytes use lexical bounds. Unsupported or null-tainted dimensions
`list[PartitionBounds]`	are omitted so the Rust layer retains their partitions.

Note

If a partition has an empty slice for a dimension, that dimension is omitted from the partition's metadata. The Rust pruning logic treats missing dimensions conservatively (never prunes on them).

`pivot(ds)` ¶

Converts an xarray Dataset to a pandas DataFrame.

`resolve_chunks(ds, chunks)` ¶

Normalise the user's chunks argument to per-dim size tuples.

Filters out keys for dims this dataset doesn't have (sub-datasets in a heterogeneous group need not contain every dimension named in the spec), then rechunks arithmetically via compute_chunks. Without a spec, a chunked dataset keeps its chunks; an unchunked one opened from storage is split along the storage chunks its backend recorded, and is one chunk along other dimensions. A dataset that records none (built in memory) is split into blocks of about _DEFAULT_BLOCK_ROWS rows.

Returns an empty mapping for scalar datasets; callers should treat that as "one block covering everything".

`resolve_table_names(ds, table_names=None, *, case_insensitive=False)` ¶

Name every dimension group of ds, honouring user overrides.

table_names maps a group's exact dimension tuple to the name its table should carry; groups it does not mention keep default_table_name. Keys that match no group in ds are ignored, so one naming map can be reused across Datasets that hold different subsets of the same variables.

Every engine adapter routes through here, which is what makes table_names={('time', 'lat', 'lon'): 'surface'} mean the same thing on DataFusion, DuckDB, and the pyarrow-dataset engines.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	The Dataset whose dimension groups are being named.	required
`table_names`	`TableNames`	User overrides; see above.	`None`
`case_insensitive`	`bool`	Fold names before comparing them for collisions — DuckDB identifiers are case-insensitive even quoted, so `'surface'` and `'SURFACE'` name the same table there even though they are distinct strings. Other engines compare case-sensitively.	`False`

Raises:

Type	Description
`ValueError`	if two groups would end up with the same name (or, with `case_insensitive=True`, names differing only in case), which would silently register one table over the other.

`shared_coord_arrays(ds)` ¶

Materialise ds's dimension coordinates once, to share.

Splitting a Dataset into per-dimension-group tables otherwise reads every shared dim coordinate once per table — a network round-trip apiece for Zarr-backed parents like ARCO-ERA5.

`ds` ¶

Reconstruct xarray Datasets from SQL query results.

The inverse of the forward Dataset-to-table pivot done by xarray_sql.df.pivot. Internally defines an XarrayDataFrame wrapper around the DataFusion DataFrame returned by Context.sql, with a XarrayDataFrame.to_dataset method that round-trips a query result back to xr.Dataset.

Reconstruction is controlled by the chunks argument to XarrayDataFrame.to_dataset -- the xarray idiom for tuning how a result is partitioned -- rather than by reflecting on the query plan:

Eager (chunks=None, or the default "inherit" when the result keeps no multi-chunk source dimension): the plan executes exactly once via execute_stream and the result is scattered into a dense in-memory Dataset. This is the right default for reductions (aggregations), whose results are small, and it never re-executes.
Lazy / chunked (chunks is a mapping, "auto", or "inherit" over a multi-chunk source dimension): data variables are backed by SQLBackendArray wrapped in xarray.core.indexing.LazilyIndexedArray and chunked via xarray's configured chunk manager (dask, cubed, ...). Each chunk maps onto the source partitions and reads its coordinate range on access by translating the indexer into a DataFusion filter expression, so only the requested partitions are materialized as Arrow RecordBatch es and scattered into numpy.

.compute() materializes the whole Dataset in memory.

`Sparsity = Literal['result', 'template']` `module-attribute` ¶

Output coordinate extent for a filtered round-trip.

"result" keeps only the dim values present in the query result, so the output is sparse and equal to whatever rows came back.
"template" reindexes to the registered Dataset's full coord ranges and fills absent cells with fill_value.

`SQLBackendArray` ¶

Bases: BackendArray

Read-only lazy N-D array view over a re-executable SQL result.

Bridges xarray's lazy-indexing interface (xarray.backends.BackendArray) to an engine query result, so an xarray Dataset can present a SQL query as if it were a materialized N-D array without actually loading any data until the caller asks for it. This is the workhorse that lets XarrayDataFrame.to_dataset (and the engine-agnostic xql.to_dataset(chunks=...)) return a Dataset cheaply.

On each __getitem__ call, the requested xarray indexer is translated into per-dimension coordinate windows and a column projection, executed through a LazyResultHandle (DataFusion, DuckDB, or Polars — each renders the windows with its own typed expression API). The resulting Arrow RecordBatch es are scattered into a preallocated numpy buffer, so only the requested data is materialized.

Constraints and caveats:

Read-only: there is no write path; the backend exists to surface query results, not to round-trip writes into a SQL store.
The underlying engine object may hold non-picklable references (DataFusion's SessionContext, a DuckDB connection). The class therefore overrides __copy__ and __deepcopy__ to return self -- this is safe because the backend is read-only.
IndexingSupport.OUTER: BasicIndexer and OuterIndexer are translated to filter predicates directly; VectorizedIndexer paths through xarray's adapter to outer-then-gather and so still works, just less efficiently.

Raises:

Type	Description
`ValueError, engine exceptions`	propagated from the underlying filter/project/execute chain if a predicate refers to a missing column, the dtype of a literal is incompatible, or the execution itself fails.
`ValueError`	from the coordinate lookup (`_CoordLookup`) when the result contains values for an irregular unique axis that are not present in the wrapper's pre-computed coord arrays -- usually a symptom of a filtered query whose coord discovery missed a value.

Constructed by _build_lazy_scan; users should not instantiate this class directly.

`dtype = np.dtype(dtype)` `instance-attribute` ¶

`shape = tuple(shape)` `instance-attribute` ¶

`XarrayDataFrame` ¶

Wrapper around a DataFusion DataFrame with xarray-aware helpers.

Returned by xarray_sql.Context.sql. Forwards every attribute it does not define itself to the wrapped DataFrame, so .collect(), .schema(), .show(), .count() all work unchanged.

Carries a private snapshot of the context's registered Datasets so to_dataset can default dims and recover metadata dropped by the forward pivot.

Users should not construct this class directly; let Context.sql produce it.

`to_dataset(dims=None, template=None, sparsity='result', fill_value=np.nan, chunks='inherit')` ¶

Convert the result to an xr.Dataset.

Parameters:

Name	Type	Description	Default
`dims`	`list[str] \| None`	Result columns to use as Dataset dimensions. When `None`, defaults to a registered Dataset's dimensions that survive into the result columns, so an aggregation that drops dims (e.g. `GROUP BY time` over a `(time, lat, lon)` grid) round-trips on the remaining dim. Raises when no dimension survives, or when several registered Datasets imply different dims (pass `dims` explicitly then).	`None`
`template`	`Dataset \| str \| None`	Source to recover metadata (attrs, encoding, non-dim coordinates, dim-coord dtype) from. Either an `xr.Dataset` used directly, or the name of a registered table (e.g. `"era5.surface"`) whose Dataset is looked up. When `None` and exactly one Dataset is registered, that one is used.	`None`
`sparsity`	`Sparsity`	`"result"` (default) keeps only dim values present in the result. `"template"` reindexes to the template's full coord ranges, filling absent cells with `fill_value`; requires a template.	`'result'`
`fill_value`	`Any`	Used when `sparsity="template"` reindexes to a wider extent. Defaults to `np.nan`.	`nan`
`chunks`	`Mapping[str, int] \| str \| None`	Output chunking, controlling laziness (an xarray idiom). `"inherit"` (default): reuse the source Dataset's chunk sizes, but only for dimensions that were genuinely split into multiple chunks in the input -- so the output chunk grid maps onto the source partitions. A reduction that drops the chunked dimension (e.g. a global aggregation) inherits nothing, and neither does a result with no resolvable source Dataset; both are returned as with `None`. `None`: lazy without a chunk manager, as with `xr.open_dataset(chunks=None)`. Each access re-executes the query narrowed to the selection and projected to the variable read, so `.load()` executes it once per variable. Dask is not required. The context must stay usable while the result is read. a mapping (e.g. `{"time": 100}`): chunk explicitly. Each chunk reads its coordinate range lazily via filter pushdown on access, through xarray's configured chunk manager (dask, cubed, ...). `"auto"`: size chunks to the chunk manager's byte target but snap boundaries to whole source partitions, so each chunk is a union of source partitions. Useful for finely partitioned sources (e.g. ERA5 `chunks={"time": 1}`), coarsening many tiny partitions into memory-sized, aligned chunks.	`'inherit'`

Returns:

Type	Description
`Dataset`	An `xr.Dataset` with `dims` as dimensions and the
`Dataset`	remaining result columns as data variables.

Raises:

Type	Description
`ValueError`	`dims` cannot be inferred, names a missing column, or the result has duplicate dim tuples; `template` names an unknown registered table; or `sparsity="template"` is requested without a resolvable template.

`to_pandas()` ¶

Materialize the result as a pd.DataFrame (DataFusion API).

`geometry` ¶

GeoArrow point-geometry columns derived from coordinate dimensions.

A regular grid's pivot already materializes per-row x/y coordinate columns; a point-geometry column is those same values under a GeoArrow extension annotation. Two encodings:

"wkb" (default) — 21-byte WKB points under the geoarrow.wkb extension name. DuckDB (>= 1.2, spatial loaded) ingests the column as a native GEOMETRY with the CRS attached, so ST_Within(geometry, ...) works with no ST_Point(x, y) construction in user SQL.
"point" — GeoArrow native points with separated coordinates (struct<x: double, y: double> under geoarrow.point): the child arrays are the coordinate columns themselves, no per-row parsing for consumers that execute on native layouts (GeoPandas 1.x, geoarrow-rs, lonboard, SedonaDB). DuckDB does not consume this encoding.

The CRS rides in the extension metadata (GeoArrow 0.2 allows authority:code strings alongside PROJJSON). OGC:CRS84 is the correct tag for plain longitude/latitude grids.

`GEOMETRY_COLUMN = 'geometry'` `module-attribute` ¶

`bbox_conjuncts(bounds, x='x', y='y', pad=0.0)` ¶

SQL bbox conjuncts for a geometry's envelope — the pruning half.

Engines do not push ST_* functions into the scan, so a geometry-only predicate reads every chunk; pairing it with range conjuncts on the coordinate columns restores pruning. This helper renders those conjuncts from a geometry's envelope::

poly = shapely.from_wkt("POLYGON (...)")
sql = (
    f"SELECT avg(risk) FROM eri "
    f"WHERE {xql.bbox_conjuncts(poly, x='x', y='y')} "
    f"AND ST_Within(geometry, ST_GeomFromText('{poly.wkt}'))"
)

Parameters:

Name	Type	Description	Default
`bounds`	`Any`	`(xmin, ymin, xmax, ymax)`, or any object with a `.bounds` attribute in that convention (shapely geometries qualify).	required
`x`	`str`	The x/longitude column name.	`'x'`
`y`	`str`	The y/latitude column name.	`'y'`
`pad`	`float`	Optional margin added on every side (e.g. to be safe around `ST_DWithin`-style predicates).	`0.0`

Returns:

Type	Description
`str`	A SQL snippet `"x" BETWEEN a AND b AND "y" BETWEEN c AND d`.

`build_geometry(encoding, x, y)` ¶

Point geometries for one batch's x/y coordinate columns.

`geometry_field(encoding, crs)` ¶

The schema field for the derived geometry column.

`lazyscan` ¶

Re-executable engine handles behind the lazy chunked round-trip.

The lazy path of to_dataset(chunks=...) re-executes the engine's query per accessed chunk, narrowed to that chunk's coordinate window and columns. That requires the engine result to be re-executable — a handle onto the query, not a one-shot stream of its rows. Each handle here adapts one engine's native lazy surface to the three operations the reconstruction needs:

schema — result column names/types, without executing the query;
distinct — one column's distinct values (coordinate discovery; the caller sorts);
fetch — the result narrowed by per-dimension windows and projected to the requested columns, as Arrow batches.

Windows are passed as DimSpec values instead of rendered SQL so each engine can express them with its own typed expression API — strings would re-open every literal-formatting pitfall (timestamps, floats, quoting) per dialect.

Handles compose with the registration seam: when the wrapped query scans a Dataset registered through xarray-sql's pushdown machinery, the per-chunk range filter flows back through the engine into XarrayPushdownDataset, so each output chunk's access reads only the source chunks it maps onto.

`DimSpec = tuple[Literal['range', 'values'], Any, Any]` `module-attribute` ¶

One dimension's window: ("range", lo, hi) (inclusive bounds; the requested coordinate positions are contiguous) or ("values", array, None) (explicit value list, for stepped/fancy indexers).

The payload stays Any: the values are coordinate scalars handed to the engine's typed expression API, which does the comparing — Python never orders them, and a concrete union over coordinate dtypes (timestamps, cftime, numerics, strings) would stay incomplete.

`DataFusionHandle` ¶

Handle over a datafusion.DataFrame.

`supports_chunked = True` `class-attribute` `instance-attribute` ¶

`distinct(column)` ¶

`fetch(specs, columns)` ¶

`schema()` ¶

`spill_parquet(path)` ¶

`stream(columns)` ¶

Execute once, yielding Arrow batches as the plan produces them.

`DuckDBHandle` ¶

Handle over a duckdb.DuckDBPyRelation.

Relations are lazy relational algebra: filter/project derive new relations and every materialization runs the query again, so a single relation can serve any number of per-chunk fetches, each narrowed to its own window — the re-executable property the module docstring requires of every handle. Predicates are built with DuckDB's typed expression API, never rendered SQL text.

Every engine call runs on one dedicated thread owned by the handle. A relation is bound to one connection, and a query over a table registered through xarray-sql re-enters Python from DuckDB's execution threads (the Arrow scan callback); driving such queries directly from several consumer threads at once (dask computing output chunks of a lazy round-trip) deadlocks between the connection's serialization, the callback's need for the GIL, and the consumer pool's own thread management. Funnelling execution through a single pre-started thread reproduces the topology that is known safe — one thread inside the engine, every other thread parked on a GIL-releasing wait.

`supports_chunked = False` `class-attribute` `instance-attribute` ¶

Chunked (lazy) reconstruction is disabled for DuckDB relations.

Windows of a chunked round-trip re-execute the relation from the consumer's worker threads (dask). A DuckDB query whose source is a Python-callback Arrow scan (any table registered through xarray-sql) intermittently deadlocks inside duckdb-python/CPython when other Python threads start or stop during execution — reproduced on duckdb 1.4-1.5 / CPython 3.12 / macOS at ~50% of runs, regardless of SET threads=1, connection serialization, or pool pre-warming. Until that upstream race is fixed, chunked DuckDB round-trips fail fast instead of hanging; the eager path (and every other handle operation) runs on one dedicated thread and is unaffected.

`distinct(column)` ¶

`fetch(specs, columns)` ¶

`schema()` ¶

`spill_parquet(path)` ¶

`stream(columns)` ¶

Execute once, yielding Arrow batches as DuckDB produces them.

The reader is closed on the engine thread, even when the consumer stops early, so the query is never torn down from another thread.

`LazyResultHandle` ¶

Bases: Protocol

A re-executable query result (see module docstring).

`supports_chunked = True` `class-attribute` `instance-attribute` ¶

Whether fetch() may be driven from consumer worker threads (the chunked reconstruction). Handles for engines that cannot safely re-execute under foreign threads set this False; the eager path remains available.

`distinct(column)` ¶

`fetch(specs, columns)` ¶

`schema()` ¶

`spill_parquet(path)` ¶

`PolarsHandle` ¶

Handle over a polars.LazyFrame.

Per-window fetches run on the streaming engine, so a window read never materializes more than the window even when the frame scans an out-of-core source.

`supports_chunked = True` `class-attribute` `instance-attribute` ¶

`distinct(column)` ¶

`fetch(specs, columns)` ¶

`schema()` ¶

`spill_parquet(path)` ¶

`stream(columns)` ¶

Execute once, yielding Arrow batches incrementally.

Unlike fetch, whose collect materializes the whole result inside the engine before any batch surfaces, this yields batches as the streaming engine produces them, so a byte budget can fire before the result is fully in memory. Requires LazyFrame.collect_batches (polars >= 1.33); older Polars raises here, before anything is collected.

`resolve_lazy_handle(result)` ¶

Adapt an engine result to a handle, or None if it is one-shot.

Recognizes DuckDB relations, Polars lazy and eager frames (an eager frame re-executes trivially over its in-memory data), and DataFusion DataFrames. pyarrow tables/readers and bare __arrow_c_stream__ objects are one-shot streams: there is no query to re-execute, so the lazy path cannot serve them.

`proj` ¶

PROJ-backed CRS transforms for SQL — the optional geo extension.

Geospatial SQL dialects expose coordinate reference system (CRS) transforms as a scalar function — PostGIS and DuckDB-spatial both call it ST_Transform — because a CRS transform is row-independent: each point's new coordinate depends only on its own old coordinate. This module brings the same capability to xarray-sql as a vectorized scalar UDF over Arrow arrays::

SELECT x, y,
       reproject(x, y, 'EPSG:32610', 'EPSG:4326')['x'] AS lon,
       reproject(x, y, 'EPSG:32610', 'EPSG:4326')['y'] AS lat
FROM grid

The CRS pair is part of the query, not baked in at registration time, so one registered UDF serves any transform — and, because the arguments are ordinary SQL expressions, the CRS may even vary per row (e.g. a CASE expression selecting the UTM zone from the longitude).

Design notes:

Both output coordinates come from one call, returned as an Arrow struct {x, y} (in always_xy order: easting/longitude first). Splitting the transform into two scalar UDFs would run PROJ twice per row and, worse, evaluate the two projections concurrently on separate expression trees.
All pyproj work runs on a dedicated pool of Python threads. DataFusion's runtime workers are not Python-created threads, and pyproj (< 3.8, see pyproj#1541) leaves a dangling PJ_CONTEXT behind when their ephemeral Python thread states are torn down, so calling pyproj in place segfaults — the UDF hands each batch to the pool instead. Pool threads are long-lived, so each caches one transformer per CRS pair (transformers must not be shared across threads), amortizing the expensive construction — PROJ database lookups and candidate-operation selection — across record batches. Concurrent partitions still transform in parallel across the pool.
Any CRS spelling pyproj.CRS accepts works: authority codes (EPSG:4326), WKT, PROJ strings (+proj=utm +zone=10), etc. An unknown CRS raises pyproj.exceptions.CRSError and fails the query loudly rather than returning wrong coordinates.
Non-finite or NULL input coordinates yield NaN output (PROJ itself would return inf); NULL CRS arguments yield NaN as well.

Requires pyproj (pip install xarray-sql[geo]). When pyproj is installed, xarray_sql.Context registers reproject() automatically; register is the explicit hook for plain DataFusion SessionContext objects or custom UDF names.

`RETURN_TYPE = pa.struct([('x', pa.float64()), ('y', pa.float64())])` `module-attribute` ¶

Arrow type returned by reproject(): destination coordinates in always_xy order — x is easting/longitude, y is northing/latitude.

`register(ctx, name='reproject')` ¶

Register the reproject(x, y, src_crs, dst_crs) scalar UDF.

Works on any DataFusion SessionContext (Context registers it automatically when pyproj is installed). The UDF returns a {x, y} struct of destination coordinates, so a query selects components with subscripts::

SELECT reproject(x, y, 'EPSG:32610', 'EPSG:4326')['x'] AS lon
FROM grid

Parameters:

Name	Type	Description	Default
`ctx`		The DataFusion session context to register the UDF on.	required
`name`	`str`	SQL name for the function (default `"reproject"`).	`'reproject'`

`reader` ¶

Lazy Arrow stream reader for xarray Datasets.

This module provides XarrayRecordBatchReader, which implements the Arrow PyCapsule Interface (arrow_c_stream) to enable zero-copy, lazy streaming of xarray data to DataFusion and other Arrow consumers.

The implementation delegates to PyArrow's RecordBatchReader for the actual stream implementation, wrapping xarray block iteration in a generator.

`XarrayRecordBatchReader` ¶

A lazy Arrow stream reader for xarray Datasets.

Implements the Arrow PyCapsule Interface (arrow_c_stream) to enable zero-copy, lazy streaming of xarray data to DataFusion and other Arrow consumers.

The key property is that xarray blocks are only converted to Arrow RecordBatches when the consumer calls get_next (e.g., during DataFusion's collect()), NOT when the reader is created or registered.

Attributes:

Name	Type	Description
`schema`	`Schema`	The Arrow schema for the stream.

Example

import xarray as xr from xarray_sql import XarrayRecordBatchReader ds = xr.tutorial.open_dataset('air_temperature') reader = XarrayRecordBatchReader(ds, chunks={'time': 240})

At this point, NO data has been read from xarray¶

Data is only read when consumed:¶

import pyarrow as pa pa_reader = pa.RecordBatchReader.from_stream(reader) for batch in pa_reader: ... print(batch.num_rows) # Data read here

`schema` `property` ¶

The Arrow schema for this stream.

`read_xarray(ds, chunks=None)` ¶

Pivots an Xarray Dataset into a PyArrow Table, partitioned by chunks.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An Xarray Dataset. All `data_vars` must share the same dimensions.	required
`chunks`	`Chunks`	Xarray-like chunks. If not provided, will default to the Dataset's chunks, or for an unchunked Dataset its storage chunks. The product of the chunk sizes becomes the standard length of each dataframe partition.	`None`

Returns:

Type	Description
`RecordBatchReader`	A PyArrow RecordBatchReader, which is a table representation of the input
`RecordBatchReader`	Dataset.

`read_xarray_table(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, coord_arrays=None, _iteration_callback=None)` ¶

Create a lazy DataFusion table from an xarray Dataset.

This is the simplest way to register xarray data with DataFusion. Data is only read when queries are executed, not during registration. The table can be queried multiple times.

Each chunk becomes a separate partition, enabling DataFusion's parallel execution across multiple cores.

Note

SQL queries with WHERE clauses on dimension columns (time, lat, lon, etc.) automatically prune partitions that can't contain matching rows — this is called filter pushdown. For example:

# This query will skip loading partitions with time < '2020-02-01'
result = ctx.sql('SELECT * FROM air WHERE time > "2020-02-01"').collect()

Supported operators: =, <, >, <=, >=, BETWEEN, IN, AND, OR.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	An xarray Dataset. All data_vars must share the same dimensions.	required
`chunks`	`Chunks`	Xarray-like chunks specification. If not provided, uses the Dataset's existing chunks, or for an unchunked Dataset its storage chunks.	`None`
`batch_size`	`int`	Maximum rows per Arrow RecordBatch emitted per partition. Smaller values let DataFusion start processing earlier; the default (65 536) works well for most datasets.	`DEFAULT_BATCH_SIZE`
`coord_arrays`	`dict[str, ndarray] \| None`	Pre-materialised coordinate arrays keyed by dim-name string. Hand in to share a single read across multiple tables built from the same parent Dataset (e.g. surface + atmosphere from ARCO-ERA5); the dim coords are otherwise read once per `read_xarray_table` call, which is a network round-trip for Zarr-backed datasets.	`None`
`_iteration_callback`	`Callable[[Block, list[str] \| None], None] \| None`	Internal callback for testing. Called with each block dict just before it's converted to Arrow.	`None`

Returns:

Type	Description
`'LazyArrowStreamTable'`	A LazyArrowStreamTable ready for registration with DataFusion.

Example

from datafusion import SessionContext import xarray as xr from xarray_sql import read_xarray_table

ds = xr.tutorial.open_dataset('air_temperature') table = read_xarray_table(ds, chunks={'time': 240})

ctx = SessionContext() ctx.register_table('air', table)

Data is only read here, during query execution¶

Filters on 'time' will prune partitions automatically!¶

result = ctx.sql('SELECT AVG(air) FROM air').collect()

`roundtrip` ¶

Engine-agnostic round-trip: Arrow query results → labeled xr.Dataset.

The second seam of xarray-sql. Any engine's result — a DuckDB relation, a pyarrow.Table, a pyarrow.RecordBatchReader, or any object implementing the Arrow PyCapsule stream protocol — plus the registered Dataset as a template is enough to rebuild a labeled, metadata-carrying Dataset. Nothing here is engine-specific: results arrive as Arrow record batches regardless of which engine executed the SQL.

Reconstruction of a re-executable result — a DuckDB relation, a Polars LazyFrame (or eager DataFrame), or a DataFusion DataFrame; see xarray_sql.lazyscan — is lazy by default: data variables are read on access by re-executing the engine's query narrowed to the selection. Passing chunks= additionally wraps them in a chunk manager, one window per chunk. DuckDB relations refuse the chunked path (a thread-safety limitation noted on DuckDBHandle); pair them with spill=True instead. One-shot Arrow streams have nothing to re-execute and are materialized once into a dense in-memory Dataset.

`to_dataset(result, dims=None, template=None, sparsity='result', fill_value=np.nan, chunks=None, coords='discover', max_result_bytes=None, spill=False)` ¶

Convert an engine's Arrow result into a labeled xr.Dataset.

The engine-agnostic counterpart of XarrayDataFrame.to_dataset: SQL in, array out, for engines xarray-sql does not wrap in a session of its own.

Example (DuckDB)::

con = duckdb.connect()
xql.register(con, "era5", ds)
rel = con.sql(
    "SELECT time, lat, lon, AVG(t2m) AS t2m FROM era5 "
    "GROUP BY time, lat, lon"
)
out = xql.to_dataset(rel, template=ds)

Parameters:

Name	Type	Description	Default
`result`	`Any`	The engine's query result: a `pyarrow.Table`, `RecordBatch` or `RecordBatchReader`, any object implementing `__arrow_c_stream__` (DuckDB relations), or an object with `fetch_record_batch()` / `to_arrow_table()`. The result is consumed once.	required
`dims`	`list[str] \| None`	Result columns to use as Dataset dimensions. When `None`, defaults to the `template`'s dimensions that survive into the result columns (so aggregations that drop dims round-trip on the remaining ones). Either `dims` or `template` must be given.	`None`
`template`	`Dataset \| None`	The source Dataset registered with the engine. Recovers metadata the tabular pivot strips (attrs, encoding, non-dim coordinates, dim-coord dtype) and provides the `dims` default.	`None`
`sparsity`	`Sparsity`	`"result"` (default) keeps only dim values present in the result. `"template"` reindexes to the template's full coord ranges, filling absent cells with `fill_value`.	`'result'`
`fill_value`	`Any`	Fill for `sparsity="template"`. Defaults to NaN.	`nan`
`chunks`	`Mapping[str, int] \| str \| None`	`None` (default) is lazy without a chunk manager, as with `xr.open_dataset(chunks=None)`: each access re-executes the query narrowed to the selection and projected to the variable read, so `.load()` executes it once per variable. Dask is not required, and DuckDB relations work, since every access runs on the calling thread. Because reads re-execute, the connection must stay open while the result is used, later changes to the source show through, and a query whose rows vary between runs (`LIMIT` without `ORDER BY`, random sampling) should be read once with `max_result_bytes=` instead. One-shot streams (Arrow tables, readers, ADBC cursors) have nothing to re-execute and are read eagerly. A mapping (e.g. `{"time": 100}`), `"auto"`, or `"inherit"` selects the lazy/chunked path: data variables are reconstructed window by window on access, each window re-executing the engine's query narrowed to its coordinate range (over a table registered through xarray-sql, that filter flows back into chunk pruning at the source). Requires a re-executable `result` — a Polars LazyFrame/DataFrame or a DataFusion DataFrame; DuckDB relations refuse the chunked path (add `spill=True`).	`None`
`coords`	`Literal['discover', 'template']`	How the lazy path learns each dimension's coordinate values. `"discover"` (default) runs one `DISTINCT` query per dim — correct for any query. `"template"` trusts the template's coord arrays instead, skipping discovery; only valid when the result spans the template's full extent (an unfiltered scan), and requires `template=`.	`'discover'`
`max_result_bytes`	`int \| None`	Optional budget for materializing the result. Setting it reads the result eagerly, in memory. Raises a clean `ValueError` (with the running size) as soon as the materializing result exceeds it — both while collecting the Arrow stream and before allocating the dense arrays — instead of exhausting memory. `None` (default) means unlimited. Results whose only surface is `to_arrow_table()` necessarily materialize in full before the budget can be checked (the check then runs on the materialized size); re-executable results stream instead, so the budget fires before full materialization.	`None`
`spill`	`bool \| str \| PathLike`	Chunked reconstruction from a one-pass on-disk spill instead of per-window re-execution: the result is streamed once (bounded memory) into a temporary Parquet file, and windows re-execute against that file. This serves the two results the re-execution path cannot — DuckDB relations and one-shot Arrow streams — and trades per-window narrowness for a single full pass plus temporary disk. `True` spills to the system temp dir; a path spills into that directory. The file is removed when the returned Dataset is garbage collected. Requires Polars; only valid with `chunks=`.	`False`

Returns: An xr.Dataset with dims as dimensions and the remaining result columns as data variables: lazily indexed by default (in memory for one-shot streams or with max_result_bytes), and lazily chunked when chunks is given.

Raises:

Type	Description
`ValueError`	When neither `dims` nor `template` resolves the dimension columns, a requested dim is missing from the result, or `sparsity="template"` is used without a template.
`TypeError`	When `result` exposes no readable Arrow stream, or `chunks` is requested for a one-shot stream that cannot be re-executed.

`sql` ¶

`XarrayContext = Context` `module-attribute` ¶

The original name of Context, kept for compatibility.

`Context` ¶

Bases: SessionContext

A datafusion SessionContext that also supports xarray.Datasets.

Also available under its original name, XarrayContext.

`from_dataset(name, input_table, *, table_names=None, chunks=None)` ¶

Register an xarray Dataset as one or more queryable SQL tables.

When all data variables share the same dimensions, the dataset is registered as a single table named name. When variables have differing dimensions (e.g. some on a 3D grid and others on a 4D grid), the dataset is split into one table per dimension group. The tables are registered under a SQL schema (namespace) named name and named <dim1>_<dim2>_... by default::

ctx.from_dataset('era5', ds, chunks={'time': 24})
# registers tables: 'era5.time_lat_lon' and
#                   'era5.time_lat_lon_level'
ctx.sql('SELECT AVG(temperature_2m) FROM era5.time_lat_lon')

Use table_names to override the name for specific dimension tuples::

ctx.from_dataset(
    'era5', ds,
    table_names={('time', 'lat', 'lon'): 'surface'},
)
ctx.sql('SELECT * FROM era5.surface')

For datasets with non-Gregorian cftime coordinates (e.g. 360_day, julian), a cftime() scalar UDF is automatically registered so you can write ergonomic SQL filters::

ctx.from_dataset("ds360", ds, chunks={"time": 6})
ctx.sql("SELECT * FROM ds360 WHERE time >= cftime('2000-07-01')")

.. note::

Only one ``cftime()`` UDF is registered per context, using the
units and calendar of the *first* non-Gregorian coordinate
encountered. If you register multiple datasets with *different*
non-Gregorian calendars (e.g. one 360_day and one julian), the
UDF from the first registration will be used for all subsequent
``cftime()`` calls and may produce incorrect offsets for the
other dataset. In that case, create a separate ``Context``
for each calendar.

Parameters:

Name	Type	Description	Default
`name`	`str`	The SQL identifier under which the dataset is registered. For datasets with uniform dimensions, this is the table name. For datasets with mixed dimensions, this is the name of a SQL schema (namespace) containing one table per dimension group.	required
`input_table`	`Dataset`	An xarray Dataset.	required
`table_names`	`TableNames`	Optional mapping from dimension tuples to custom table names within the schema, used when the dataset has variables with differing dimensions.	`None`
`chunks`	`Chunks`	Xarray-like chunks specification. If not provided, uses the Dataset's existing chunks, or for an unchunked Dataset its storage chunks (see register).	`None`

Returns:

Type	Description
	self, to allow chaining.

`sql(query, *args, **kwargs)` ¶

Run a SQL query, returning an XarrayDataFrame wrapper.

Identical to datafusion.SessionContext.sql except the returned object wraps the DataFusion DataFrame. The wrapper exposes .to_pandas() (unchanged), forwards every other DataFusion method via __getattr__, and adds .to_dataset(dimension_columns=[...]) for round-tripping the result back to an xr.Dataset.

Parameters:

Name	Type	Description	Default
`query`	`str`	A SQL query string.	required
`*args`		Forwarded to `SessionContext.sql`.	`()`
`**kwargs`		Forwarded to `SessionContext.sql`.	`{}`

Returns:

Type	Description
`XarrayDataFrame`	An XarrayDataFrame wrapping the DataFusion DataFrame.

xarray-sql¶

xarray_sql ¶

XarrayContext = Context module-attribute ¶

__all__ = ['cftime', 'Context', 'XarrayContext', 'FlightSQLServer', 'read_xarray_table', 'read_xarray', 'arrow_dataset', 'arrow_datasets', 'bbox_conjuncts', 'register', 'serve', 'to_dataset', 'from_map'] module-attribute ¶

Context ¶

from_dataset(name, input_table, *, table_names=None, chunks=None) ¶

sql(query, *args, **kwargs) ¶

FlightSQLServer ¶

is_running property ¶

port property ¶

uri property ¶

register(name, ds, *, chunks=None, table_names=None) ¶

serve(host='127.0.0.1', port=0) ¶

shutdown(timeout=5.0) ¶

wait(poll_interval=0.5) ¶

__getattr__(name) ¶

arrow_dataset(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, prefetch=DEFAULT_PREFETCH, prefetch_bytes=None, coalesce_rows=None, geometry=None, geometry_encoding='wkb', geometry_crs='OGC:CRS84') ¶

arrow_datasets(ds, name=None, *, chunks=None, table_names=None, **kwargs) ¶

bbox_conjuncts(bounds, x='x', y='y', pad=0.0) ¶

from_map(func, *iterables, args=None, **kwargs) ¶

read_xarray(ds, chunks=None) ¶

read_xarray_table(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, coord_arrays=None, _iteration_callback=None) ¶

Data is only read here, during query execution¶

Filters on 'time' will prune partitions automatically!¶

register(con, name, ds, *, chunks=None, table_names=None, **kwargs) ¶

serve(datasets, host='127.0.0.1', port=0, *, chunks=None, table_names=None, memory_limit=None) ¶

to_dataset(result, dims=None, template=None, sparsity='result', fill_value=np.nan, chunks=None, coords='discover', max_result_bytes=None, spill=False) ¶

backends ¶

EngineAdapter ¶

matches(con) staticmethod ¶

register(con, name, ds, *, chunks=None, table_names=None, **kwargs) staticmethod ¶

FlightSQLServer ¶

is_running property ¶

port property ¶

uri property ¶

register(name, ds, *, chunks=None, table_names=None) ¶

serve(host='127.0.0.1', port=0) ¶

shutdown(timeout=5.0) ¶

wait(poll_interval=0.5) ¶

XarrayArrowStream ¶

XarrayPushdownDataset ¶

partition_expression property ¶

schema property ¶

count_rows(filter=None, **kwargs) ¶

filter(expression) ¶

get_fragments(filter=None) ¶

join(*args, **kwargs) ¶

join_asof(*args, **kwargs) ¶

replace_schema(schema) ¶

scanner(columns=None, filter=None, batch_size=None, **kwargs) ¶

sort_by(sorting, **kwargs) ¶

arrow_dataset(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, prefetch=DEFAULT_PREFETCH, prefetch_bytes=None, coalesce_rows=None, geometry=None, geometry_encoding='wkb', geometry_crs='OGC:CRS84') ¶

arrow_datasets(ds, name=None, *, chunks=None, table_names=None, **kwargs) ¶

get_adapter(con) ¶

register(con, name, ds, *, chunks=None, table_names=None, **kwargs) ¶

register_adapter(cls) ¶

serve(datasets, host='127.0.0.1', port=0, *, chunks=None, table_names=None, memory_limit=None) ¶

adbc ¶

ADBCAdapter ¶

matches(con) staticmethod ¶

register(con, name, ds, *, chunks=None, table_names=None, mode='create', temporary=False, ingest_options=None, **kwargs) staticmethod ¶

base ¶

ConT = TypeVar('ConT') module-attribute ¶

EngineAdapter ¶

matches(con) staticmethod ¶

register(con, name, ds, *, chunks=None, table_names=None, **kwargs) staticmethod ¶

get_adapter(con) ¶

register(con, name, ds, *, chunks=None, table_names=None, **kwargs) ¶

register_adapter(cls) ¶

datafusion ¶

DataFusionAdapter ¶

matches(con) staticmethod ¶

register(con, name, ds, *, chunks=None, table_names=None, **kwargs) staticmethod ¶

duckdb ¶

DuckDBAdapter ¶

matches(con) staticmethod ¶

register(con, name, ds, *, chunks=None, table_names=None, **kwargs) staticmethod ¶

XarrayArrowStream ¶

XarrayPushdownDataset ¶

partition_expression property ¶

`xarray_sql` ¶

`XarrayContext = Context` `module-attribute` ¶

`all = ['cftime', 'Context', 'XarrayContext', 'FlightSQLServer', 'read_xarray_table', 'read_xarray', 'arrow_dataset', 'arrow_datasets', 'bbox_conjuncts', 'register', 'serve', 'to_dataset', 'from_map']` `module-attribute` ¶

`Context` ¶

`from_dataset(name, input_table, *, table_names=None, chunks=None)` ¶

`sql(query, *args, **kwargs)` ¶

`FlightSQLServer` ¶

`is_running` `property` ¶

`port` `property` ¶

`uri` `property` ¶

`register(name, ds, *, chunks=None, table_names=None)` ¶

`serve(host='127.0.0.1', port=0)` ¶

`shutdown(timeout=5.0)` ¶

`wait(poll_interval=0.5)` ¶

`getattr(name)` ¶

`arrow_dataset(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, prefetch=DEFAULT_PREFETCH, prefetch_bytes=None, coalesce_rows=None, geometry=None, geometry_encoding='wkb', geometry_crs='OGC:CRS84')` ¶

`arrow_datasets(ds, name=None, *, chunks=None, table_names=None, **kwargs)` ¶

`bbox_conjuncts(bounds, x='x', y='y', pad=0.0)` ¶

`from_map(func, *iterables, args=None, **kwargs)` ¶

`read_xarray(ds, chunks=None)` ¶

`read_xarray_table(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, coord_arrays=None, _iteration_callback=None)` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` ¶

`serve(datasets, host='127.0.0.1', port=0, *, chunks=None, table_names=None, memory_limit=None)` ¶

`to_dataset(result, dims=None, template=None, sparsity='result', fill_value=np.nan, chunks=None, coords='discover', max_result_bytes=None, spill=False)` ¶

`backends` ¶

`EngineAdapter` ¶

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

`FlightSQLServer` ¶

`is_running` `property` ¶

`port` `property` ¶

`uri` `property` ¶

`register(name, ds, *, chunks=None, table_names=None)` ¶

`serve(host='127.0.0.1', port=0)` ¶

`shutdown(timeout=5.0)` ¶

`wait(poll_interval=0.5)` ¶

`XarrayArrowStream` ¶

`XarrayPushdownDataset` ¶

`partition_expression` `property` ¶

`schema` `property` ¶

`count_rows(filter=None, **kwargs)` ¶

`filter(expression)` ¶

`get_fragments(filter=None)` ¶

`join(*args, **kwargs)` ¶

`join_asof(*args, **kwargs)` ¶

`replace_schema(schema)` ¶

`scanner(columns=None, filter=None, batch_size=None, **kwargs)` ¶

`sort_by(sorting, **kwargs)` ¶

`arrow_dataset(ds, chunks=None, *, batch_size=DEFAULT_BATCH_SIZE, prefetch=DEFAULT_PREFETCH, prefetch_bytes=None, coalesce_rows=None, geometry=None, geometry_encoding='wkb', geometry_crs='OGC:CRS84')` ¶

`arrow_datasets(ds, name=None, *, chunks=None, table_names=None, **kwargs)` ¶

`get_adapter(con)` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` ¶

`register_adapter(cls)` ¶

`serve(datasets, host='127.0.0.1', port=0, *, chunks=None, table_names=None, memory_limit=None)` ¶

`adbc` ¶

`ADBCAdapter` ¶

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, mode='create', temporary=False, ingest_options=None, **kwargs)` `staticmethod` ¶

`base` ¶

`ConT = TypeVar('ConT')` `module-attribute` ¶

`EngineAdapter` ¶

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

`get_adapter(con)` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` ¶

`register_adapter(cls)` ¶

`datafusion` ¶

`DataFusionAdapter` ¶

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

`duckdb` ¶

`DuckDBAdapter` ¶

`matches(con)` `staticmethod` ¶

`register(con, name, ds, *, chunks=None, table_names=None, **kwargs)` `staticmethod` ¶

`XarrayArrowStream` ¶

`XarrayPushdownDataset` ¶

`partition_expression` `property` ¶

`schema` `property` ¶

`count_rows(filter=None, **kwargs)` ¶

`filter(expression)` ¶