polybase

Reference

Commands &
HTTP API.

Exact arguments, fields, responses and defaults for the tools that ship in the evaluation preview.

The four tools and the launcher

Run commands from the extracted package directory. Paths beginning with ./bin/ select the supplied executable. All four tools accept --help, -h, --version and -V without opening a database or starting a service. Their application arguments are positional.

Included executables
ToolInvocationPurpose
polybase./bin/polybase COMMAND DIR …Open one local database, run a command, then close it.
polybase-console./bin/polybase-console EXISTING_DB_DIR [PORT]Own one local database and serve its browser console and HTTP API. Default port: 7340.
polybase-node./bin/polybase-node CONFIG_JSON NODE_ID EXISTING_EMPTY_OR_REPLICA_DIR [--lab-controls]Run one voter from the fixed three-node configuration.
polybase-cluster-console./bin/polybase-cluster-console CONFIG_JSON [PORT]Serve the cluster browser console and HTTP API. Default port: 7342.

The database tools do not require Rust, Node, Docker or a cloud account. The optional launcher needs Python 3.10 or later:

python3 scripts/start-local-cluster.py ./my-lab --bin-dir ./bin
Launcher arguments
ArgumentDefaultMeaning
directoryRequiredLab directory. A new unbound directory must be empty. Existing configuration and node directories are retained.
--bin-dirSource-build location; pass ./bin for the downloadDirectory containing the node and cluster-console executables.
--console-port7342Port for the cluster console.
--node-ports7351 7352 7353Three distinct ports for a new lab. Existing configuration determines existing voter addresses.

The launcher accepts four distinct ports between 1024 and 65535. It starts detached processes and records logs and observed process IDs. Closing its console leaves the nodes running. See how to start and stop your lab.

Every local CLI command

The CLI accepts UTF-8 strings as keys and values. Quote spaces and JSON in your shell. It does not parse JSON documents. Binary values are best read through the console's hex editor or the HTTP interface; CLI output converts non-UTF-8 bytes to display text.

Local command reference
CommandArgumentsResult and behavior
initDIRCreate the directory if its parent exists, or open an existing directory. Print initialized head=N. Existing data is recovered, never reset.
getDIR KEYPrint the current value plus a newline, or (absent). An empty stored value prints an empty line.
putDIR KEY VALUEInsert or replace one byte value. Print committed N.
deleteDIR KEYSubmit a deletion. Print its commit position. A raw write can create a commit even when the key was already absent.
batchDIR (put KEY VALUE | delete KEY)…Commit supplied mutations together, in supplied order. Invalid syntax submits no partial batch.
snapshotDIRPrint snapshot N, then current key/value entries in byte-key order. This is display output, not a backup format.
eventsDIR AFTER LIMITRead raw committed batches strictly after AFTER. LIMIT is an integer from 1 through 1024. Print commits, mutations and a next=N head=H footer.
./bin/polybase init ./workspace-db
./bin/polybase batch ./workspace-db put tasks/1 open put notes/1 'A first note'
./bin/polybase get ./workspace-db notes/1
./bin/polybase batch ./workspace-db put tasks/1 done delete notes/1
./bin/polybase events ./workspace-db 0 10
./bin/polybase snapshot ./workspace-db

For this fresh database, the two batches print committed 1 and committed 2; the last snapshot starts with snapshot 2. Commands return exit status 0 on success and 1 on an error. Only init creates a missing directory; valid commands can initialize an existing empty directory when opening it. Stop the console before running the CLI against its directory.

Local HTTP conventions

Start polybase-console and use the exact printed address, normally http://127.0.0.1:7340. These are the preview console's local endpoints, not a hosted SaaS API. They are useful for evaluation scripts; there is no separately distributed SDK.

./bin/polybase-console ./workspace-db 7340
# From a second terminal:
curl --http1.1 http://127.0.0.1:7340/api/health
curl --http1.1 -H 'Content-Type: application/json' \
  -H 'Origin: http://127.0.0.1:7340' \
  --data '{"key":"7461736b732f31"}' \
  http://127.0.0.1:7340/api/read

POST requests require exactly Content-Type: application/json and an Origin matching the console address, including its port. Use 127.0.0.1 consistently rather than substituting localhost. The HTTP/1.1 interface accepts ordinary Content-Length bodies; streamed bodies and Expect headers are unsupported. Each response closes the connection.

Wire value types
TypeRepresentationExample
Keys and valuesUse lowercase, even-length hexadecimal strings representing bytes. Responses use lowercase hex. An empty string represents zero bytes.tasks/1 → 7461736b732f31; open → 6f70656e.
Absent values / deletionsJSON null. This differs from the empty byte value "".{"key":"7461736b732f31","value":null}
Positions and session IDsDecimal strings, not JavaScript floating-point numbers."0", "42". Keep large positions as strings or arbitrary-precision integers.
Query and request tokensOpaque strings returned by the server. Preserve the complete string.Use the returned token or request_token; do not construct one.
InstanceOpaque string identifying this running console process.Pair the returned instance with the session id.

Unknown fields are rejected in the typed request bodies below. Always include required fields, even when their value is an empty string or null. JSON is just one possible value encoding: the server compares the encoded bytes you supply.

Complete local endpoint reference

Local HTTP endpoints
Method and pathRequest bodyResponse / effect
GET /api/healthNoneCurrent local position, retained floor, identity, instance, state and subscription counters. Does not create a commit.
POST /api/read{key}{value, position}. Value is hex or null. The read is validated at the returned local prefix.
POST /api/query{key_prefix, predicate, value}{id, instance, token, position, rows}. Rows contain hex key and value. Registers a filtered snapshot and later changes.
POST /api/resume{token}{id, instance}. Registers continuation from a saved token; does not send another snapshot.
POST /api/poll{id, instance, token}{commits, token, position, head, floor}. Scans up to 64 retained commits. Apply complete changes before saving the returned token.
POST /api/unsubscribe{id, instance, token}{closed:true}. Releases the matching query registration.
POST /api/request{}{request_token}. Issue a named-write identity. Save it with the exact intended request before submitting.
POST /api/commit{reads, mutations, request_token?}One conditional atomic batch. A named success returns {position,wrote:true,durable:true,replayed}. Without a request token there is no retained duplicate-request receipt.
POST /api/outcome{token}{position}, with a decimal string for a retained outcome or null for no outcome observed.
POST /api/checkpoint{}{generation,floor}. Full checkpoint, retaining the existing history floor.
POST /api/checkpoint-incremental{}{generation,floor}. Optional incremental checkpoint; may use the full path when required by the current state or caps.
POST /api/compact{position}{generation,floor}. Explicitly advance retained history through the supplied decimal position, protecting active subscriber cursors.
POST /api/close{}{closed:true}. Stops the local console and releases its database owner after workers finish.

Query fields

key_prefix is a hex-encoded byte prefix. Use 7461736b732f for tasks/, or "" for all keys. predicate is exactly any, equals or prefix. value is a hex string; supply "" with any. Equals compares the entire byte value, and prefix compares its starting bytes. Neither reads fields inside JSON.

curl --http1.1 -H 'Content-Type: application/json' \
  -H 'Origin: http://127.0.0.1:7340' \
  --data '{"key_prefix":"7461736b732f","predicate":"any","value":""}' \
  http://127.0.0.1:7340/api/query

Conditional write fields

reads is an array of expected entries, each containing key and value. Use null to expect an absent key. There must be one unique expected key for every mutated key. Extra expected reads may check another dependency. mutations is a nonempty array of {key,value}: a hex string inserts or replaces; null deletes. Mutation order is retained, including repeated keys. The local console accepts at most 128 expected reads and 128 mutations per request.

A condition checks bytes, not a revision number. If a value changes away and later returns to the same bytes before your submission, that condition can still succeed. See the condition model. Use a named request for an outcome you can inspect after a missing reply. The working Python example constructs a complete named request without placeholder tokens.

Reading responses correctly

A query response establishes the starting result. Poll responses contain complete transaction groups. Each group has a decimal position and a changes array. Each change has key, before and after. Null before means an insertion; null after means a deletion. A replacement has both values.

One transaction yields at most one net change per matching key, even if it writes that key several times. A whole-value no-op can produce no query change, while the cursor still advances. Empty commits therefore does not mean the returned token should be ignored. head is the prefix captured for that response, and floor is the oldest retained position.

Useful health fields
FieldTypeMeaning
head, floorDecimal stringsCurrent local committed prefix and oldest resumable prefix.
checkpoint, generationDecimal stringsPosition selected in the recovery checkpoint and selected storage generation.
identity, instanceOpaque stringsPersistent database identity and running-console identity. A restart changes instance; replacing the database changes identity.
keys, wal_bytesNonnegative JSON numbersLive keys and bytes in the selected write-ahead log.
version_bytesNonnegative JSON numberLogical charge for concurrent retained versions, rather than total process memory.
subscribers, subscription_bytesNonnegative JSON numbersCurrent subscriber count and aggregate logical subscription charge.
work_cache_bytes, work_hits, work_missesNonnegative JSON numbersShared query-projection cache charge and reuse counters.
checkpoint_layersNonnegative JSON numberLayers in the currently selected full/incremental checkpoint chain.

These counters are observations, not one coordinated application snapshot or a promise about memory use. Use a query or validated read for record values.

Status codes and your next action

Local errors contain {code,message}. Check both the HTTP status and the structured code. A timeout or broken connection after submission may hide a successful write; keep the original named request.

Local response errors
HTTP / codeMeaningNext action
400 invalid_requestUnsupported route, field, shape, number or byte encoding.Correct the request. An invalid batch is not partly submitted.
400 invalid_tokenToken is malformed, from another history, ahead of the head, or incompatible with the query.Use the token belonging to this database and exact query; do not edit its bytes.
403 origin_rejectedHost/Origin does not match the exact printed loopback address.Use the same 127.0.0.1 address and port in the URL and Origin header.
409 conflictAn expected value changed; this local request did not commit.Read the latest values and explicitly decide on new intent. A failed retry does not resolve an earlier uncertain attempt.
409 request_mismatchA retained request token was reused with different intent.Inspect and preserve the original payload; do not change a submitted token's intent.
409 history_pinnedRequested compaction would pass an active query cursor.Poll or close the subscriber before choosing a new maintenance boundary.
410 history_expiredRequired query history is below the retained floor.Start a fresh query and replace its result and cursor together.
410 subscription_expiredThe registration was idle, fell behind, or belongs to a prior console instance.Resume the saved result/token if history remains, or obtain a fresh snapshot.
410 request_expiredThe named request's retained horizon ended.Reconcile with application state/history. Do not automatically issue a new request identity.
413 request_too_largeHTTP body exceeds 128 KiB.Reduce the request. Remember that hex roughly doubles the underlying byte payload.
429 capacityA bounded queue, result, state or history allowance cannot admit this work.Read the message; reduce query scope or batch size, release idle queries, or perform appropriate explicit maintenance.
503 uncertainWrite/maintenance outcome may be unknown, or the instance cannot continue after an earlier failure.Keep exact intent; stop and reopen when required, then inspect its named outcome.
500 engine_errorAnother engine error, including storage failures needing inspection.Keep the directory intact, read the message and inspect the owning process. Do not assume a missing reply means no write.

Defaults exposed by the preview

The supplied command-line servers use their default options. There are no CLI flags for increasing these limits. Counts and logical bytes bound individual data structures; they are not hard process-memory or timing budgets.

Local data and query bounds
ResourceDefaultImplication
Encoded local transaction1 MiB / 1,024 mutationsIncludes encoding overhead. Console requests have the smaller HTTP/128-mutation limits.
Key64 KiBCount UTF-8 bytes, not characters. Empty byte keys are supported.
Live local state64 MiB key+value payload / 100,000 keysAll collection prefixes share the same database limits.
Retained local WAL256 MiBNew writes reject at the cap; old history is not automatically removed.
Console request / response128 KiB / 8 MiBHex and JSON representation add overhead.
Console query registrations32 / 60 seconds idleResume or start a fresh query after idle expiry.
Query result4,096 rows / 2 MiB logical chargeA broad prefix can fail rather than return a silently truncated result.
Query lag / poll4,096 commits / up to 64 scanned commits per HTTP pollKeep polling; smaller results do not remove the retained-history horizon.
Named local receipts4,096 / 16 MiB logical chargeWhole canonical requests are retained, not just token strings.
Displayed browser changes200 changesDisplay history is bounded; it is not the database's retained WAL.

The bundled GUARANTEES.md lists further engine bounds for versions, leases, pages and checkpoints. The implemented scope explains the practical limits.

The separate cluster-console API

The cluster console normally runs at http://127.0.0.1:7342. Every API call is POST and requires node as a decimal string, read_mode as quorum or local, and an optional minimum of null or {index,term} with decimal strings. The same exact Origin and JSON conventions apply.

curl --http1.1 -H 'Content-Type: application/json' \
  -H 'Origin: http://127.0.0.1:7342' \
  --data '{"node":"1","read_mode":"local","minimum":null}' \
  http://127.0.0.1:7342/api/health

The health call reports all configured nodes, their availability and roles, plus the selected context. It does not promise a quorum-confirmed record read. Select the observed leader explicitly for writes and quorum reads; do not assume node 1 is the leader. Node IDs are exactly 1, 2 and 3, with distinct IPv4 loopback addresses and one nonzero 16-byte hex partition identity in cluster.json. Keep that binding intact on restart.

Cluster route differences
RouteBody in addition to contextImportant difference
/api/healthNo additional fieldsPOST, including explicit context. Returns node status, selected availability and this console instance.
/api/query{key_prefix,predicate,value}Returns snapshot rows, a term-bound position/token, registration id and instance.
/api/resume{token}Checks the selected node/read mode, then returns id and instance without another snapshot.
/api/poll, /api/unsubscribe{id,instance,token}Session and token belong to this partition/query. A valid cursor may continue on another eligible node.
/api/read{key}Returns value, position, term and mode. Local reads can be stale; a retained minimum can reject a node that is behind.
/api/requestNo additional fieldsRequest issuance first confirms a quorum on the selected leader.
/api/commit{reads,mutations,request_token}Named token is required. At most 64 expected keys and 64 mutations. Check the durable receipt's status, wrote, position and term.
/api/outcome{token,intent:{reads,mutations}}The original canonical intent is required. Outcome lookup uses the selected node's quorum-confirmed behavior.
/api/checkpoint{confirm:"COMPACT"}Select a physical snapshot and compact this node. Older query cursors can expire; this is not the local checkpoint's history-preserving behavior.
/api/lab{action,confirm}With lab controls enabled, action is isolate, heal, campaign or stop; confirmation is that action in uppercase. These are explicit experiments on the selected owned node.
/api/close{confirm:"CLOSE CONSOLE"}Close only the console; nodes continue running.

/api/checkpoint-incremental and /api/compact are local-console routes and are not cluster-console routes. Replicated commits can return HTTP 200 with a durable non-write status such as conflict or capacity. A successful write has status=applied, wrote=true and durable=true; read the receipt rather than treating HTTP 200 alone as a mutation.

Additional cluster errors include 409 not_leader, 409 behind, 503 quorum_unavailable and 502 protocol. Keep the last applied result/cursor when a read cannot satisfy its chosen mode. A leader hint is a selection hint, not permission to replace a pending write's identity. See original-request reconciliation.

Replicated lab bounds
ResourceDefault
Application state512 KiB / 4,096 keys
Application command128 KiB / up to 64 mutations and 64 expected keys
Retained protocol entries512 / 4 MiB, including entries that do not change application records
Retained request outcomes256 / 512 KiB canonical intent plus receipt charge
Snapshot transfer2 MiB, one incoming transfer, 64 KiB chunks
TopologyExactly three voters, one partition, one host in the supplied lab
Choose the right interface

The local and cluster histories have different identities, position formats, request horizons and maintenance behavior. A local database directory is not a replica's inner store, and local tokens cannot be used as cluster tokens.