Write - TopK

Documents in TopK are JSON-like objects composed of key-value pairs. Each document within a collection:

Must include a unique _id field
Must conform to the schema defined for the collection

Fields defined in the schema can be indexed for vector, keyword, or other retrieval strategies.

Upsert documents

To upsert documents, pass a list of documents to the upsert() function:

client.collection("books").upsert(
    [
        {
            "_id": "book-1",
            "title": "The Great Gatsby",
            "published_year": 1925,
            "title_embedding": [0.12, 0.67, 0.82, 0.53, ...]
        },
        {
            "_id": "book-2",
            "title": "To Kill a Mockingbird",
            "published_year": 1960,
            "title_embedding": [0.42, 0.53, 0.65, 0.33, ...]
        },
        {
            "_id": "book-3",
            "title": "1984",
            "published_year": 1949,
            "title_embedding": [0.59, 0.33, 0.71, 0.61, ...]
        }
    ]
)

await client.collection("books").upsert([
  {
    _id: "book-1",
    title: "The Great Gatsby",
    published_year: 1925,
    title_embedding: [0.12, 0.67, 0.82, 0.53],
  },
  {
    _id: "book-2",
    title: "To Kill a Mockingbird",
    published_year: 1960,
    title_embedding: [0.42, 0.53, 0.65, 0.33],
  },
  {
    _id: "book-3",
    title: "1984",
    published_year: 1949,
    title_embedding: [0.59, 0.33, 0.71, 0.61],
  },
]);

-- INSERT has upsert semantics: an existing document with the same _id is replaced
INSERT INTO books (_id, title, published_year, title_embedding)
VALUES
  ('book-1', 'The Great Gatsby',        1925, '[0.12, 0.67, 0.82, 0.53]'::f32_vector),
  ('book-2', 'To Kill a Mockingbird',   1960, '[0.42, 0.53, 0.65, 0.33]'::f32_vector),
  ('book-3', '1984',                    1949, '[0.59, 0.33, 0.71, 0.61]'::f32_vector);

Every document must have a string _id field.
If a document with the specified _id doesn’t exist, a new document will be inserted.
If a document with the same _id already exists, the existing document will be replaced with the new one.

The upsert() function does not perform a partial update or merge - the entire document is being replaced.

Each document you send is serialized as a Protocol Buffers (protobuf) message. The encoded size of that message must be 128KB or smaller.

Additional (non-schema) fields

You may include fields that are not defined in the collection schema. These fields:

Are stored with the document
Can be returned to the client in query results
Can be used for filtering in queries

Fields that are not defined in the schema are not indexed. If you want to use a field for vector search, semantic search, keyword search or multi-vector search, it must be declared in the schema and have a corresponding index defined.

from topk_sdk.schema import text, int, f32_vector, vector_index, keyword_index
from topk_sdk.query import select, field, fn

client.collections().create(
    "books",
    schema={
        "title": text().index(keyword_index()).required(),
        "published_year": int().required(),
        "title_embedding": f32_vector(dimension=1024).index(vector_index(metric="cosine")).required(),
    },
)

client.collection("books").upsert([
    {
        "_id": "book-1",
        "title": "The Great Gatsby",
        "published_year": 1925,
        "title_embedding": [0.12, 0.67, 0.82, 0.53, ...],
        "tags": ["fiction", "classic"], # non-schema field
        "source_url": "https://example.com/gatsby", # non-schema field
    }
])

client.collection("books").query(
    select(
        "title",
        "source_url",
        "title_similarity": fn.semantic_similarity("title", "classic American novel"),
    )
    .filter(field("tags").contains("fiction")) # non-schema fields can still be used for filtering
)

import { text, int, f32Vector, vectorIndex, keywordIndex } from "topk-js/schema";
import { select, field, fn } from "topk-js/query";

await client.collections().create("books", {
  title: text().index(keywordIndex()).required(),
  published_year: int().required(),
  title_embedding: f32Vector({ dimension: 1024 })
    .index(vectorIndex({ metric: "cosine" }))
    .required()
});

await client.collection("books").upsert([
  {
    _id: "book-1",
    title: "The Great Gatsby",
    published_year: 1925,
    title_embedding: [0.12, 0.67, 0.82, 0.53],
    tags: ["fiction", "classic"], // non-schema field
    source_url: "https://example.com/gatsby", // non-schema field
  },
]);

const docs = await client.collection("books").query(
  select({
    title: field("title"),
    source_url: field("source_url"),
    title_similarity: fn.semanticSimilarity("title", "classic American novel"),
  })
  .filter(field("tags").contains("fiction")) // non-schema fields can still be used for filtering
);

Supported types

TopK documents are a flat structure of key-value pairs. The following value types are supported:

Type	Python Type	JavaScript Type	Helper Function
String	`str`	`string`	-
Integer	`int`	`number`	-
Float	`float`	`number`	-
Boolean	`bool`	`boolean`	-
String list	`list[str]`	`string[]`	`string_list()`
F32 list	`list[float]`	`number[]`	`f32_list()`
F64 list	use helper	use helper	`f64_list()`
I32 list	use helper	use helper	`i32_list()`
I64 list	use helper	use helper	`i64_list()`
U32 list	use helper	use helper	`u32_list()`
F8 vector	use helper	use helper	`f8_vector()`
F16 vector	use helper	use helper	`f16_vector()`
F32 vector	`list[float]`	`number[]`	`f32_vector()`
U8 vector	use helper	use helper	`u8_vector()`
I8 vector	use helper	use helper	`i8_vector()`
Binary vector	use helper	use helper	`binary_vector()`
F32 sparse vector	use helper	use helper	`f32_sparse_vector()`
U8 sparse vector	use helper	use helper	`u8_sparse_vector()`
Matrix	use helper	use helper	`matrix()`
Bytes	use helper	use helper	`bytes()`
Struct	`dict[str, Any]`	`Record<string, any>`	`struct()`

Delete documents

You can delete documents by their _id or using a filter expression. Both methods provide the same consistency guarantees and will be reflected in query/get results according to your consistency level.

Delete documents by `_id`

To delete documents by their _id, pass a list of _ids to the delete() method:

client.collection("books").delete(
    ["book-1", "book-2", "book-3"]
)

await client.collection("books").delete(["book-1", "book-2", "book-3"]);

DELETE FROM books WHERE _id IN ('book-1', 'book-2', 'book-3');

The delete() method returns an LSN (Log Sequence Number) that you can pass to subsequent queries for read-after-write consistency.

Delete documents by filter expression

To delete documents that match a predicate, you can pass a filter expression to the delete() function.

from topk_sdk.query import field

client.collection("books").delete(field("published_year").lt(1997))

import { field } from "topk-js/query";

await client.collection("books").delete(field("published_year").lt(1997));

DELETE FROM books WHERE published_year < 1997;

Passing a filter expression to delete() is useful for deleting documents that match a specific condition, such as all documents for a specific tenant e.g. field("_id").starts_with("tenant-123/").

​Upsert documents

​Additional (non-schema) fields

​Supported types

​Delete documents

​Delete documents by _id

​Delete documents by filter expression

Upsert documents

Additional (non-schema) fields

Supported types

Delete documents

Delete documents by `_id`

Delete documents by filter expression