How-to guides

How to work
with your database.

Practical steps for your records, collections, query results, retained history and local evaluation lab.

How to install and open the preview

Choose the Mac or Linux package on the download page. Mac requires Apple Silicon and macOS 15 or later; Linux requires x86-64 and glibc 2.35 or later. Extract the package into a directory you can write. The four executable tools live in bin/.

1. Download the matching checksum file alongside the package. Replace PACKAGE below with the filename you downloaded.
2. Check it on Mac with shasum -a 256 -c PACKAGE.tar.gz.sha256, or on Linux with sha256sum -c PACKAGE.tar.gz.sha256.
3. Extract it with tar -xzf PACKAGE.tar.gz and change into the resulting polybase-… directory.
4. Run the included version/help commands, then create a new database directory.

./bin/polybase --version
./bin/polybase --help
./bin/polybase init ./workspace-db
./bin/polybase-console ./workspace-db

For a fresh directory, init prints initialized head=0. Open http://127.0.0.1:7340 in your browser. The console owns this directory until you stop it. Use Ctrl-C in its terminal, or its confirmed Close engine… action, before using the CLI on that directory.

If the parent directory does not exist, create that parent first. The supplied init command creates one database directory rather than a missing parent tree. Reopening an existing directory recovers its records; init is not a reset command.

Mac executables use an ad-hoc signature rather than notarization. If macOS asks you to allow the download, follow Apple's Open Anyway instructions for that executable. The preview does not require changing a global system setting for other applications. For a different console port, use ./bin/polybase-console ./workspace-db 7341 and the matching printed address.

How to create, replace and delete records

With the console stopped, CLI commands open and close the database themselves. A put inserts a missing key or replaces its whole value. A delete removes its current entry. Use a batch when several keys belong to one change.

./bin/polybase put ./workspace-db tasks/1 open
./bin/polybase put ./workspace-db notes/1 'A first note'
./bin/polybase get ./workspace-db notes/1
./bin/polybase batch ./workspace-db put tasks/1 done delete notes/1
./bin/polybase get ./workspace-db tasks/1
./bin/polybase get ./workspace-db notes/1

The last two gets print done and (absent). The batch replaces the task and deletes the note together. Its commit number depends on your existing history. It does not compare values you observed earlier; use console staging or a conditional HTTP write for that check.

In the browser, click a record's Edit button, change its value, then click Stage mutation. Inspect the staged list and explicitly click Commit transaction. To delete, choose Delete key before staging. Staging alone does not change database records; Discard all clears local staged edits.

A stale edit reports a conflict. Refresh the current record and decide what to change; do not silently overwrite the latest observation. If a request was already submitted and its outcome is uncertain, use the outcome controls before starting a new request.

How to choose your own collection names

Pick a stable prefix and use the full key to identify each record. No create-collection step is needed. A note, task and measurement can share one local database while retaining different complete keys.

./bin/polybase batch ./workspace-db \
  put tasks/1 '{"title":"Write a note","done":false}' \
  put notes/1 'A first note' \
  put sensors/room-a '{"temperature":21.5,"unit":"C"}'
./bin/polybase snapshot ./workspace-db

Open the console, enter tasks/ in Collection / key prefix, select Any value, and start the query. Change the prefix to notes/ or sensors/ to see those records. Clear it to see every prefix. Include the slash when you want the collection boundary: tasks/ excludes tasks-archive/.

For nested conventions, keys such as notes/project-a/1 and notes/project-b/1 support queries for all notes or one chosen project's prefix. This is still a byte-prefix match; there is no directory object or enforced hierarchy in the database. A collection becomes empty when it has no matching live records.

For a small collection, query its prefix and stage each matching key's deletion. All staged deletes can commit together within the console's 128-mutation and request-size bounds. There is no bulk drop-collection command. Do not delete database files to remove a logical collection.

How to store JSON or exact binary bytes

Use single quotes around JSON in a POSIX shell so its inner double quotes remain part of the value. The server stores that exact JSON encoding. Read it back and parse it in your application when you need document fields.

./bin/polybase put ./workspace-db settings/theme '{"name":"paper","contrast":"normal"}'
./bin/polybase get ./workspace-db settings/theme

The get prints the JSON you supplied. Whitespace differences remain byte differences, so choose a consistent serialization when expected values or whole-value queries should compare equal. Polybase does not add timestamps, generate IDs or validate your application fields for you.

For binary data in the browser, select Hexadecimal. Enter a hexadecimal key and value with no 0x prefix or spaces. For example, blobs/1 is 626c6f62732f31, and 00ff0a stores three bytes: zero, 255 and newline. Stage and commit, then edit the record again in hex mode to verify those exact bytes.

An empty text/hex value is a real zero-byte value. A deletion is a separate mutation. The HTTP API uses null for an absent value or deletion, and the empty hex string for zero bytes. CLI get/snapshot output is readable text and can be lossy for binary data; it should not be used as a binary export format.

How to query a collection or a whole-value condition

Starting a different query obtains a fresh snapshot and token. An equals condition on open will not find {"status":"open"}, because that JSON is a different complete value. To filter document fields, read and parse records in your application or maintain an application-owned view prefix.

A broad query can exceed the 4,096-row or 2 MiB logical-result limit. The console reports the limit rather than silently returning a truncated collection. Choose a narrower key grouping. The CLI snapshot command can inspect the bounded whole local state while the console is stopped, but it is a display command, not a paginated query or backup.

Maintain an explicit view across collections

For a small evaluation example, store a task and its open-view entry in one batch. When it completes, update the task and remove the view entry together. Your application owns this convention and must keep it consistent.

./bin/polybase batch ./workspace-db \
  put tasks/view-demo '{"title":"Try a view","done":false}' \
  put views/open-tasks/view-demo '{"title":"Try a view","done":false}'
./bin/polybase batch ./workspace-db \
  put tasks/view-demo '{"title":"Try a view","done":true}' \
  delete views/open-tasks/view-demo

A query for views/open-tasks/ then excludes the completed task. These raw CLI batches demonstrate the representation, not concurrent read validation. In an HTTP application, include expected values for both representations when its write depends on them.

How to follow and resume live changes

For a simple raw event inspection, stop the console, take a CLI snapshot and note its printed position. Read events strictly after that position. Use each event page's next position for the following page, while respecting its captured head.

./bin/polybase snapshot ./workspace-db
# If that snapshot says position 4:
./bin/polybase events ./workspace-db 4 64

For a filtered live result, use a native query in the console or /api/query. Keep its rows and token together. Poll with the query's id, instance and last applied token. Apply every complete commit group, validate each before-value against your saved result, then save the new result and token as one application state.

1. Preserve the current applied result when a request fails.
2. If the console restarts or its temporary registration expires, send /api/resume with the saved token.
3. Use the returned id and instance with the original saved result/token to poll. Resume does not send another snapshot.
4. If retained history has expired, start a fresh query and replace its whole result and token together.

A lost poll response can cause repeated delivery when you retry from the previous applied token. Advancing a token without applying its changes loses application progress; saving a token without the matching result is also insufficient. The Python tutorial shows one result/cursor file and a named cross-collection write.

How to resolve an uncertain write

In the browser, an uncertain write keeps the original staged intent and disables ordinary fresh commits. Click Inspect request outcome. If a retained outcome is found, the console can reconcile that original request. If you explicitly choose Retry original request, it reuses the saved token and exact intent.

For an HTTP client, save the issued token and complete reads/mutations body before sending /api/commit. Inspect /api/outcome using that token after a lost reply. A retained identical retry returns its original position without another application mutation event.

Raw CLI writes have no named identity, so their result after a lost response is an application reconciliation task. Avoid automatically rerunning a raw batch that may already have committed. In the lab, save the original intent as well as the token because its outcome endpoint requires both.

How to checkpoint and manage retained history

Use the local console's full or incremental checkpoint action to select a recovery representation while preserving retained history. Check the selected generation and checkpoint position in the Storage section or health response. An incremental action may fall back to a full checkpoint at its bounds.

Before compaction, identify the oldest query cursor and named request your application still needs. Advance or close subscribers appropriately. Enter a valid local position in Compact through position and invoke the action. The local console protects active subscriber cursors; a pinned-history response asks you to resolve that dependency first.

curl --http1.1 -H 'Content-Type: application/json' \
  -H 'Origin: http://127.0.0.1:7340' --data '{}' \
  http://127.0.0.1:7340/api/checkpoint

To compact from a script, read the current head/floor, choose a boundary based on application retention needs, and POST {"position":"N"} to /api/compact. It is an application decision; the guide does not choose N on your behalf. Older tokens can expire when the retained floor passes their required history.

Checkpointing alone does not necessarily free retained WAL history. There is no automatic background cleanup or flag that converts a full history into an unbounded one. When a data or history cap is reached, read the specific capacity message and select the relevant maintenance or narrower work.

How to back up and reopen a database

1. Stop the console and any other owner of the directory. For the lab, stop every owner whose replica directory you copy.
2. Copy the entire closed directory into a new backup location, including its identity, manifest, selected generations and retained history. Do not copy only a WAL or snapshot display.
3. Keep the original closed if you open its copy. Writable copies of one history identity are not independent live databases.
4. Open the copied local database and inspect records/events to verify the closed-directory copy.

# After stopping the database owner; choose a new backup path.
cp -a ./workspace-db ./workspace-db-backup
./bin/polybase snapshot ./workspace-db-backup
./bin/polybase get ./workspace-db-backup tasks/1

This example assumes the backup destination does not already exist. Opening validates the selected history and can complete normal recovery work. Keep the original and copied histories from being independently written at the same time. There is no online backup or automatic repair command.

For a replica, the outer binding file and complete inner store are one unit. Do not restore an older voter directory into an active partition, change the voter identity or replace missing bound storage with an empty directory. The lab's full configuration and node histories must remain coherent. See local and replicated history identities.

How to run and stop the three-node research lab

The lab uses a separate directory from your local database. From the extracted package, start it with the supplied launcher. It creates voters 1–3, one partition identity and a browser console, preserving an existing lab on later runs.

python3 scripts/start-local-cluster.py ./my-lab --bin-dir ./bin
# Open http://127.0.0.1:7342

For a new lab with other ports:

python3 scripts/start-local-cluster.py ./another-lab --bin-dir ./bin \
  --node-ports 7451 7452 7453 --console-port 7442

Choose the observed leader for writes and quorum-confirmed reads. Stage task and note keys together and inspect the durable outcome. Local follower reads are explicit and can be stale; an index/term minimum can reject a follower that has not reached the required prefix. An unavailable quorum reports an error rather than silently switching your read to local mode.

Closing the cluster console stops only that console. Detached node processes remain running. The launcher prints observed process IDs and records logs under the chosen lab directory. Inspect a process's current command and lab path before stopping it; an old PID file does not establish which process currently owns that number. Prefer separate foreground terminals when learning restart behavior:

# After stopping the existing owners; use the retained configuration.
./bin/polybase-node ./my-lab/cluster.json 1 ./my-lab/1 --lab-controls
./bin/polybase-node ./my-lab/cluster.json 2 ./my-lab/2 --lab-controls
./bin/polybase-node ./my-lab/cluster.json 3 ./my-lab/3 --lab-controls
./bin/polybase-cluster-console ./my-lab/cluster.json 7342

Run each command in its own terminal; Ctrl-C stops that terminal's process. Do not open a replica's inner store with the local console. Missing node directories, occupied ports or an inconsistent binding require inspection, not a reset of voting history.

The lab's checkpoint action snapshots and physically compacts the selected node. That can expire older query cursors, while still-valid request outcomes remain retained. This differs from a local history-preserving checkpoint. The lab has exactly three voters and one partition; the supplied setup is for local evaluation rather than a hosted or multi-host deployment.

How to identify and resolve common problems

Useful details when reporting an evaluation issue are the tool's version, target/OS, exact command or API code, whether the directory is local or a replica, and the sequence of steps using synthetic records. Do not send your real database contents to the public website; there is no upload or issue-submission feature here.