CLI - TopK

Command-line interface for TopK.

Installation

brew tap topk-io/topk
brew install topk

Quick start

topk login
topk import ./books.parquet [--project-id <PROJECT_ID>] --region aws-us-east-1-elastica
topk import postgres://user:pw@host/db 'public.*' [--project-id <PROJECT_ID>] --region aws-us-east-1-elastica

topk import discovers a schema, shows the plan, and imports on confirmation — see import.

Authentication

To authenticate, run:

topk login

Commands

topk login

If the CLI cannot open your browser automatically, or you prefer to open it manually, use topk login --no-browser to print the login URL.

logout

Remove the saved session:

topk logout

project

list

List projects in the current organization.

topk project list

topk project list marks the currently selected project with *.

select

Select a default project.

topk project select  # interactive picker
topk project select <PROJECT_ID>

clear

Clear the selected project:

topk project clear

current

Show the currently selected project.

topk project current

get

Get a project by ID.

topk project get <PROJECT_ID>

argument	description
`<PROJECT_ID>`	ID of the project

create

Create a project.

topk project create <NAME>

argument	description
`<NAME>`	Display name of the new project

delete

Delete a project by ID.

topk project delete <PROJECT_ID> [-y]

argument	description
`<PROJECT_ID>`	ID of the project
`-y`, `--yes`	Skip the confirmation prompt; required when stdin is not a terminal

region

list

List available regions.

topk region list

sql

Create and manage collections, read and write documents, and run keyword, vector, and semantic searches with TopK SQL.

topk sql <QUERY|--file <FILE>> [OPTIONS]

argument	description
`[QUERY]`	SQL to run, or `-` to read it from stdin; required unless `--file` is given
`-f`, `--file <FILE>`	Read SQL from a file, or `-` for stdin
`--region <REGION>`	Region to connect to (env `TOPK_REGION`); required. See https://docs.topk.io/regions
`--project-id <PROJECT_ID>`	Project to access with your login (env `TOPK_PROJECT_ID`); defaults to the project selected with `topk project select`
`--api-key <API_KEY>`	Use an API key instead of your login; overrides the `TOPK_API_KEY` environment variable

Run a query

Log in and find your project ID, then run SQL against a collection. This example searches a title field with a semantic index:

topk login
topk project list
topk sql "SELECT _id, title, semantic_similarity(title, 'classic American novel') AS score
          FROM books ORDER BY score DESC LIMIT 10" \
  --project-id <project-id> --region aws-us-east-1-elastica

Alternatively, pass --api-key or set TOPK_API_KEY instead of using your login and --project-id.

See the TopK SQL reference for complete documentation.

Run SQL from a file or stdin

topk sql -f queries.sql --project-id <project-id> --region aws-us-east-1-elastica
echo "SELECT COUNT(*) FROM books" | topk sql - --project-id <project-id> --region aws-us-east-1-elastica

Inspect tables

psql-style commands list and describe tables:

topk sql '\dt' --project-id <project-id> --region aws-us-east-1-elastica          # all tables
topk sql '\dt books*' --project-id <project-id> --region aws-us-east-1-elastica   # matching a pattern
topk sql '\d books' --project-id <project-id> --region aws-us-east-1-elastica     # columns and indexes
topk sql '\di' --project-id <project-id> --region aws-us-east-1-elastica          # all indexes
topk sql '\dA' --project-id <project-id> --region aws-us-east-1-elastica          # index kinds
topk sql '\dT' --project-id <project-id> --region aws-us-east-1-elastica          # column types
topk sql '\?' --project-id <project-id> --region aws-us-east-1-elastica           # supported commands

Output

Results print as one table per statement by default, with row counts on stderr. Use -o json to stream one JSON object per row as it arrives, for example topk sql … -o json | jq. Pipe long output to a pager yourself, for example | less.

import

Bulk import into TopK collections. Every run prints the plan as a TOML spec and asks before writing; collections are created right after.

topk import [SOURCE] [OBJECTS]... [OPTIONS]

argument	description
`[SOURCE]`	Source URL, file path, or glob; also required with `--spec` or `--resume`, except for files
`[OBJECTS]...`	Source objects to import; exact names, globs, or `<object>=<collection>`
`--region <REGION>`	Region to import into (env `TOPK_REGION`); required. See https://docs.topk.io/regions
`--to <COLLECTION>`	Name the target collection; required when it cannot be derived
`--id <COLUMN>`	Column to use as the document id (`_id`); use when it can’t be auto-detected
`--dry-run`	Print the spec and a sample of documents, without importing
`-f`, `--spec <SPEC>`	Read a TOML import spec
`-y`, `--yes`	Skip the confirmation prompt; required when stdin is not a terminal
`--resume <RUN>`	Continue a run that stopped, by the id in its header; `-f` swaps in an edited spec
`--filter <FILTER>`	Only read rows matching a filter, in the source’s language: SQL `WHERE` for files and databases, a JSON query for mongodb/elasticsearch
`--limit <LIMIT>`	Read at most this many rows per object
`--partition <COLUMN>`	Column whose value names each row’s partition; see Split rows across partitions
`--continue-on-error`	Skip documents that fail instead of stopping; exits non-zero if any did
`-c`, `--concurrency <N>`	Concurrent upserts in flight, budgeted across the whole run (default `16`)
`--batch-bytes <SIZE>`	Bytes of documents per upsert (default `8MiB`)
`--project-id <PROJECT_ID>`	Project to import into with your login (env `TOPK_PROJECT_ID`); defaults to the project selected with `topk project select`
`--api-key <API_KEY>`	Use an API key instead of your login; overrides the `TOPK_API_KEY` environment variable

Authenticate

topk login
topk project list
topk import ./books.parquet --project-id <project-id> --region aws-us-east-1-elastica

Set TOPK_PROJECT_ID and TOPK_REGION to omit the corresponding flags. Explicit flags override environment variables.

Use topk project list to find your project ID and topk region list to list regions. Access tokens are cached and renewed automatically. Run topk project select once to omit --project-id from later commands. An API key (--api-key or TOPK_API_KEY) takes precedence over the selected project. Alternatively, pass --api-key or set TOPK_API_KEY instead of logging in and providing --project-id. Get an API key in the TopK console.

--project-id or TOPK_PROJECT_ID cannot be combined with an API key (--api-key or TOPK_API_KEY). Unset TOPK_API_KEY to use your login.

Import a database

topk import postgres://user:pw@host/db 'public.*'
topk import mysql://root@host/shop orders
topk import mongodb://host/shop products
topk import es+https://user:pw@es.example.com 'products*'
topk import postgres://host/db orders --filter "created_at > '2024-01-01'"

Objects are exact names, globs, or <object>=<collection> renames.

Import files

topk import ./books.parquet                                   # .csv .tsv .json(l) .arrow .avro
topk import 's3://bucket/books/*.parquet' --to books          # a glob needs --to
topk import gs://bucket/books.csv                             # az:// hf:// http(s):// too

A single file names its collection.

Copy a TopK collection

topk import topk://aws-us-east-1-elastica/books --to books-v2                        # reindex under a new schema
topk import topk://aws-us-east-1-elastica/books --region aws-eu-central-1-monstera   # copy us → eu

Schema and indexes copy as-is; the copy is additive.

Split rows across partitions

topk import ./events.parquet --partition tenant               # one partition per `tenant` value

Each row goes to the partition named by that column, so one source stream writes into as many partitions as the column has values.

The column only has to exist in the source. It does not have to be an imported field.

Preview and edit the plan

topk import postgres://host/db orders --dry-run > spec.toml   # spec on stdout, sample documents on stderr
vim spec.toml                                                 # drop columns, fix types, add indexes
topk import "$DB_URL" -f spec.toml --yes                      # run

Keep credentials on the command line or in the environment, never in the spec — the spec is meant to go in git.

Resume a stopped run

Stop a run — ^C, a lost connection — and pick it up where it left off:

topk import postgres://host/db --resume 01J9...               # run id printed at start

Resume skips finished collections and continues unfinished ones from their source checkpoints, after all preceding partition writes succeeded. Limits include rows consumed before the checkpoint, including skipped rows. Without --resume, a re-run re-imports everything — upserts are idempotent.

Sources

source	url	filter	auth
postgres	`postgres://`	SQL `WHERE`	in-URL, `PGPASSWORD`
mysql	`mysql://`	SQL `WHERE`	in-URL, `MYSQL_PWD`
sqlite	`sqlite:<path>`	SQL `WHERE`	—
files	path, glob, `s3://` `r2://` `gs://` `az://` `hf://` `http(s)://`	SQL `WHERE`	credential chain; `hf auth login`; http anonymous
elasticsearch	`es://` `es+https://`, `*.cloud.es.io`	query DSL	in-URL, `ELASTIC_API_KEY` / `ELASTIC_PASSWORD`
mongodb	`mongodb://`	find document	in-URL, `MONGODB_URI`
topk	`topk://[<key>@]<region>/<collection>`	—	URL key, else the run’s API key or access token

Spec

One TOML table per collection — from, id, filter/partition/limit as the flags (partition names a column), and a fields whitelist (only declared fields import):

[books]
from = "public.books"
id = "sku"

[books.fields]
title = { type = "text", index = "semantic" }
year = { type = "int", from = "published_year" }
embedding = { type = "f32_vector", dim = 768, index = { vector = { metric = "cosine" } } }

field key
`type`	`text` `int` `float` `bool` `bytes` `timestamp` `struct` `_list` `_vector` `_matrix` `_sparse_vector`
`from`	source column, when the name differs
`required`	fail the document if missing
`truncate`	max chars, text only
`dim` / `cols`	vector dimension / matrix row width
`index`	`"keyword"` `"exact"` `"semantic"` `"ngram"` `{ vector = { metric = "cosine" \| "euclidean" \| "dot_product" \| "hamming" } }` `{ multi_vector = {} }`

Discovery takes the source’s types; declaring a different type converts. Embeddings that arrive as float lists, text ("[1,2,3]") or packed bytes become vectors by declaring f32_vector and dim (f32_matrix and cols for multi-vectors); a sparse vector reads numeric keys or parallel indices/values lists; epoch numbers or date text become timestamp. Conversions are exact or the document fails; a narrower declaration (truncate included) is how loss is accepted.

Failures and limits

A document fails when a value does not convert exactly, its id is missing or empty, or it is over 200 KB. A failure stops the run, ready to --resume; --continue-on-error skips them instead.

there is no schema migration: an existing collection whose schema differs is rejected
an upsert replaces the whole document, so a spec that omits an existing field clears it

Global flags

These flags are accepted by every command:

`--output`

Options:

text (default)
json

Updating the CLI

To update CLI to the latest version, run:

brew update
brew upgrade topk

​Installation

​Quick start

​Authentication

​Commands

​login

​logout

​project

​list

​select

​clear

​current

​get

​create

​delete

​region

​list

​sql

​Run a query

​Run SQL from a file or stdin

​Inspect tables

​Output

​import

​Authenticate

​Import a database

​Import files

​Copy a TopK collection

​Split rows across partitions

​Preview and edit the plan

​Resume a stopped run

​Sources

​Spec

​Failures and limits

​Global flags

​--output

​Updating the CLI

Installation

Quick start

Authentication

Commands

login

logout

project

list

select

clear

current

get

create

delete

region

list

sql

Run a query

Run SQL from a file or stdin

Inspect tables

Output

import

Authenticate

Import a database

Import files

Copy a TopK collection

Split rows across partitions

Preview and edit the plan

Resume a stopped run

Sources

Spec

Failures and limits

Global flags

`--output`

Updating the CLI