Explanation

Collections,
commits & change.

Understand the model before choosing collection names, write boundaries, query cursors and retention rules.

When should you choose Polybase?

Evaluate Polybase when your application needs both current records and a dependable way to follow their changes. Its local database supplies atomic writes, a snapshot with a matching change cursor, filtered result changes and retained outcomes for named writes. Those pieces share one committed history, so your application can work with a defined protocol instead of designing every boundary itself.

For example, a workspace updates a task and a note together. A task-list client takes a snapshot, pauses, then catches up without showing half of that transaction. If the write reply was lost, the writer checks the original request. Polybase provides these operations for your own byte-key collections. You still save the client result and cursor together and choose how much history to retain.

What Polybase supplies directly

• A snapshot and its later changes. A query returns matching records at position S and a token for later changes. The filter uses byte-key ranges and whole-value predicates. It does not inspect JSON fields.
• Complete transaction groups. Query changes describe net before/after transitions per matching key. Several changes in one commit stay together, including entries entering or leaving the result. Raw commit events remain available separately.
• The outcome of your original write. A named request binds its token to the exact expected reads and ordered mutations. Within retained history, an identical retry returns the original outcome; altered intent rejects.
• Explicit history boundaries. A cursor binds its database history and query. An older cursor can expire after compaction; your application receives an error and decides whether to take a fresh snapshot.

These are convenience and integration reasons to evaluate the product. Atomic transactions, durable logs and change notification also exist in other databases. Polybase's appeal is how its supplied operations fit together for a local application that follows changes.

Choose SQLite when SQL is the main requirement

SQLite is a strong starting point for embedded local storage, relational queries and reporting. If your application needs joins, indexes or flexible SQL queries, those capabilities are outside Polybase's preview. See SQLite's own guide to appropriate uses.

You can design an SQLite application to update its records and append rows to a change-log table in the same SQL transaction. Its session extension also captures table changes for later application elsewhere. SQLite is therefore a viable foundation for applications that track changes.

With an application-managed log, you define the snapshot position, transaction grouping, changes to a query result, saved cursor, retention and rules for retrying writes. Existing libraries may already do that work. Evaluate Polybase if its supplied contract reduces the protocol your application needs to own; keep SQLite when its query capabilities and your current change-handling approach serve you better.

Choose FoundationDB for a distributed transactional foundation

FoundationDB provides a distributed ordered key-value database with ACID transactions and strict serializability. Its transaction guide describes the concurrency model. Its documented key watches report that a value changed; reading after a notification does not promise every intervening value.

An application or layer can build transaction logs, projections and request deduplication on that foundation. A key watch by itself has a different job from Polybase's protocol for resuming a filtered query result.

Polybase's downloadable runtime is a local evaluation, with a separate three-voter research lab on one host and one fixed partition. That lab does not establish availability after losing the host or supply FoundationDB's operating model. Choose FoundationDB when distributed transactions are your requirement; evaluate Polybase when the local application contract is the part you want to explore.

A key-value store with a change feed may already be enough

Change data capture, often shortened to CDC, exposes changes from stored data so other components can follow them. Implementations vary. A feed derived from the database's own committed log can already keep state and changes together. For a concrete example, etcd documents ordered, resumable watches, transaction boundaries and a history window bounded by compaction. Polybase does not have an exclusive claim to these ideas.

Compare the exact store and connector you would use. Ask these questions about the complete application path:

• Does the initial snapshot include a position from which changes can resume without a gap?
• Are all relevant changes from one transaction delivered together?
• Do you receive raw writes, key changes or net transitions for a filtered result, including records entering and leaving it?
• What identifies the saved cursor, and how does the system report that its history expired?
• After a missing write reply, how do you determine the original request's outcome without creating a new intent?

A stack that already answers these questions may be the right choice. Polybase packages one particular set of answers into its local interfaces. It can reduce integration work when that set matches your application; moving an established application just to obtain another change feed may add work.

Try the boundary your application actually needs

Start with the runnable live-workspace example. It writes across tasks and notes, replays the original named request and resumes a saved task result. Then replace those prefixes and values with a disposable model of your own workflow. Evaluate the snapshot, changes, missing-reply and retention behavior before deciding whether this contract helps.

The preview is limited to byte-range and whole-value queries, bounded state and history, supported local filesystems and the published Mac/Linux tools. It includes no public engine SDK, general SQL layer or production service agreement. The binary license permits evaluation and development; production use requires different terms. See the actual limits and evaluation terms.

There are no like-for-like benchmarks here showing Polybase faster or more durable than SQLite, FoundationDB or another store. Choose an evaluation for the application contract; choose a production system using the capabilities, measurements and support your workload needs. Hosted Polybase SaaS is coming soon.

A record is a byte key and a byte value

A task tracker, notebook and measurement collector can all use Polybase. The engine stores keys and values without requiring fields such as a title, status, owner or timestamp. Your application chooses the record shape. A key names one current record; putting the same key replaces its value, and deleting the key removes it from current state.

Collections organize that key space using a prefix you choose. tasks/1 and notes/1 have the same short record ID but different complete keys, so they are independent records. Nothing needs to register tasks before the first write. Collection names are not limited to the examples in this guide.

A slash has no built-in hierarchy. It is a useful separator because a query for tasks/ includes tasks/1 and tasks/archive/1, while excluding tasks-archive/1. Querying tasks without the slash also matches that similarly named archive prefix. Choose the complete prefix for your intended grouping.

An empty prefix matches every key. Keys sort by their bytes, not by locale, numbers or dates inside a value. For example, readings/10 sorts before readings/2. If ordering matters, your application can use a fixed-width ID such as readings/000002. Different casing and differently encoded Unicode strings remain different byte keys; normalize names in your application when you need that convention.

Text, JSON and binary use the same storage model

The CLI's arguments encode UTF-8 text. The browser editor lets you select UTF-8 or hexadecimal. The local HTTP API represents keys and values as hexadecimal strings so it can carry arbitrary bytes without changing their meaning.

JSON is convenient for application records, but Polybase stores the bytes you supply rather than parsing fields. Whitespace, property order and number spelling can make two JSON encodings different byte values. A query's equals predicate compares the entire encoded value. A prefix predicate compares its starting bytes. Neither can ask whether done is false or temperature exceeds 20.

Your application can parse a JSON value after reading it. For a frequently used view, you can maintain an explicit prefix such as views/open-tasks/ alongside the primary task record. This is an application-managed view: write both representations in one batch and remove the view entry when a task no longer belongs. Polybase does not maintain that view automatically.

tasks/1             → {"title":"Write a note","done":false}
notes/1             → A first note
views/open-tasks/1   → {"title":"Write a note","done":false}

There is no imposed JSON schema, document validation, automatic field index, SQL parser or general query planner in this preview. The freedom to choose an encoding also leaves your application responsible for validating records and coordinating changes to its own derived views.

One transaction publishes all its changes together

Updating a task and its note in separate writes can leave a half-finished relationship if the application stops between them. A batch puts the related mutations in one transaction. A successful commit publishes every mutation together, including changes to keys in different collection prefixes within that database.

stage keys and values
        ↓
one atomic committed batch
        ↓
current state   +   a replayable commit event

Mutations apply in the supplied order. If one batch writes the same key more than once, current state contains its final value. Raw commit events retain the mutation batch; query-result changes report only the final net transition for that key. Other normal API operations do not see half the batch.

A local commit succeeds after writing its transaction records, synchronizing the write-ahead log and durable frontier, selecting that frontier through a rename, synchronizing the directory and publishing the state. The write-ahead log, usually called WAL, is the committed history used for recovery and raw events. The frontier records the protected committed boundary.

These operations ask the operating system and filesystem to persist data. Their honoring synchronization is an assumption. Tests that stop a process establish process/recovery behavior; they do not independently prove sudden hardware power-loss survival. The database stops accepting normal work after a write failure that leaves the result uncertain, so reopening can inspect the selected durable history.

Expected values decide whether your edit is still current

Suppose you read a task, change its title locally and submit the result after another writer already changed that task. An expected value lets the server compare your original observation with the current record before applying the batch. If a required expected value differs, the local conditional request rejects without applying its other mutations.

The console automatically stages the value you observed, including null for an absent key. Its commit sends expected values for every changed key. In an HTTP client, supply these explicitly in reads. Include another expected key when your write depends on that record too. Conditions can cover different collection prefixes, and the whole request succeeds or rejects together.

The comparison is against the complete bytes, not a record revision. A value that changes away and returns to the same bytes can satisfy the condition. The lower-level concurrent Rust database implements additional point/range transaction validation, but those Rust transaction sessions are not exposed as a public downloaded SDK or as HTTP transaction-session endpoints.

Raw CLI put and batch commands are unconditional write batches. They do not carry these expected reads, and they do not assign a named request identity. Use the console or its local HTTP API when evaluating expected-value edits and retained write outcomes.

A snapshot and its later changes share one boundary

A local snapshot at position S includes the committed prefix through S. Reading events strictly after S avoids a gap between obtaining the initial state and following changes. A native query applies this boundary to one filtered result: it returns matching rows with a token, then later polls return complete transaction groups strictly after that token.

A change describes optional before and after values. Before absent / after present is an insert. Both present is a replacement. After absent is a deletion. If a value no longer matches the query's whole-value predicate, the result sees a removal even if the underlying key still exists. If it starts matching, the result sees an insertion.

Several mutations in one commit remain one query change group. Repeated writes to the same key coalesce to its net result transition. An equal-value write can leave the query result unchanged. A commit outside the query's collection also leaves it unchanged. The query token still advances past that work, even when the response contains no visible changes.

A query token belongs to one persistent history and exact query. Keep it opaque. Save the result and its last applied token together, after applying the whole response. Resuming from an old token can repeat changes that your application already received. Resuming does not return a second snapshot, so the token alone is insufficient to reconstruct your application's saved view.

A query registration ID and instance identify a running console's temporary session. They differ from the persistent history token. After a console restart, resume with your saved result/token to obtain a new session, or start a fresh query and replace both. After changing the query, take a fresh snapshot rather than attaching the old cursor to a different filter.

A missing write reply leaves an outcome to inspect

A server can commit a request and then lose the connection before your client receives the acknowledgment. Resubmitting those mutations as a new request could perform them again. Named requests provide an identity for the original intent within retained history.

Issue a request token, then save it with the exact expected reads and ordered mutations before submitting. An identical retained retry returns the original commit position rather than creating a second state transition or event. Reusing that token with altered intent rejects. Outcome lookup reports the retained result, including after reopening or a supported leader change in the lab.

An absent lookup is an observation at that moment. It does not cancel an in-flight submission or prove that one can never commit later. A request that has expired cannot safely be turned into a fresh request automatically. Inspect application state/history and decide how to reconcile the intent. This is a bounded original-request protocol, not a promise that every network reply arrives once or that external effects such as sending email happen once.

The browser console saves pending intent in this tab's session storage. That helps after a reload; it is not a database-backed client journal, a cross-tab shared state store or a guarantee that browser storage survives every kind of exit. Evaluation scripts should save their own exact pending request before sending it.

Checkpoint and compaction solve different problems

A checkpoint writes a selected recovery representation of state while preserving the retained history floor. Reopening applies only mutations after the checkpoint's covered position, while still checking retained history. Checkpointing therefore does not make every old cursor expire and does not necessarily reduce all retained log bytes.

Compaction explicitly advances the oldest retained position. It preserves committed current state and sequence numbers, but older queries or requests can lose the history they need. Local console compaction protects active subscriber cursors and reports a pinned-history error rather than silently passing them. Close or advance a subscriber before choosing an appropriate boundary.

The incremental checkpoint is an optional local experiment. It can reuse earlier layers and write a bounded delta, with a full-checkpoint fallback at its caps. It does not implement background compaction or guarantee that reopening is faster for every workload. Both kinds of maintenance use the same selected-generation recovery rules.

History and state limits reject new work rather than silently erasing old committed data. When the WAL reaches its cap, a history-preserving checkpoint alone may not free enough retained history. Choose a compaction boundary based on the cursors and named requests your application still needs. No automatic retention scheduler makes that application decision for you.

One local database and one replicated partition have separate contracts

A majority acknowledgment does not say every follower already applied the write. Select read mode and node deliberately. An explicit minimum index/term can reject a follower that is behind, but retained evidence can itself expire. The lab's three nodes share one host and filesystem, so this setup does not demonstrate availability after losing that host.

The lab does not convert native local Rust MVCC sessions into distributed transactions. Its application batches use committed expected-byte conditions in one fixed partition. There is no cross-partition transaction order, membership replacement or sharding in the supplied preview.

What the evaluation establishes

The implemented preview supports atomic byte-key batches, strict recovery, commit events, local concurrent validation, native snapshot/delta queries, named request outcomes and explicit maintenance. Its separate fixed-partition research runtime adds majority commits, leader changes and bounded snapshot catch-up.

It does not include a general document-field query engine, SQL, automatic schemas, automatic indexes, online backup, automatic repair, a hosted account or a production service agreement. The supported download targets are Apple Silicon macOS 15+ and x86-64 Linux with glibc 2.35+. Use a supported local filesystem and disposable evaluation records.

Keep the complete directory intact when a stored-file error is reported. Recovery does not invent missing selected metadata or skip damaged committed records to make a database appear healthy. Back up a closed directory as one unit. The bundled GUARANTEES.md contains the full contracts and bounds.

Polybase SaaS is coming soon. The downloadable evaluation and website do not provide that hosted service today. You can evaluate collections, record encodings and application change handling locally while that hosted product is being prepared.