Ingest

To run queries in TopK, your documents need to be uploaded. Once a document is uploaded, TopK parses, chunks, and indexes its content to power Ask, Search, and Research.

Uploading documents

Documents are uploaded to a dataset. A dataset is a named container for your documents. Each document in a dataset is identified by a document ID.

Learn how to create your dataset here.

CLI
Python SDK
JavaScript SDK

# Upload all supported files in the directory
topk upload ./docs --dataset my-docs

# Upload all supported files in the directory and subdirectories recursively
topk upload ./docs -r --dataset my-docs

# Upload all supported files in the directory by glob pattern
topk upload "./docs/*" --dataset my-docs

# Upload all supported files in the directory by glob pattern recursively
topk upload "./docs/**/*" -r --dataset my-docs

# Upload all PDFs in the directory by glob pattern
topk upload "./docs/*.pdf" --dataset my-docs

# Upload all PDFs in the directory by glob pattern recursively
topk upload "./docs/**/*.pdf" --dataset my-docs

# Upload a single file
topk upload ./report.pdf --dataset my-docs

import os
from pathlib import Path
from topk_sdk import Client

client = Client(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

handle = client.dataset("my-docs").upsert_file(
    "report-2024",                       # document ID
    Path("./report.pdf"),                # path to file
    {"title": "Annual Report 2024"},     # document metadata (optional)
)

print(handle)  # e.g. "hdl_abc123"

import { Client } from "topk-js";

const client = new Client({
  apiKey: process.env.TOPK_API_KEY!,
  region: process.env.TOPK_REGION!,
});

const handle = await client.dataset("my-docs").upsertFile(
  "report-2024",                    // document ID
  { path: "./report.pdf" },         // path to file
  { title: "Annual Report 2024" },  // document metadata (optional)
);

console.log(handle);  // e.g. "hdl_abc123"

Supported formats

Format	MIME type	Extensions
PDF	`application/pdf`	`.pdf`
Markdown	`text/markdown`	`.md`
HTML	`text/html`	`.html`
PNG	`image/png`	`.png`
JPEG	`image/jpeg`	`.jpg`, `.jpeg`
GIF	`image/gif`	`.gif`
WebP	`image/webp`	`.webp`
TIFF	`image/tiff`	`.tiff`, `.tif`
BMP	`image/bmp`	`.bmp`

Attaching metadata

Each document can be associated with its own metadata. You can attach additional information about the document, such as title, author, date, or category.

Python SDK
JavaScript SDK

from pathlib import Path

resp = client.dataset("my-docs").upsert_file(
    "report-2024",
    Path("./report.pdf"),
    {
        "title": "Annual Report 2024",
        "author": "Finance Team",
        "year": 2024,
        "category": "financials",
        "tags": ["annual-report", "investor-relations"],
        "categories": ["finance", "public-company"],
    },
)

const resp = await client.dataset("my-docs").upsertFile(
  "report-2024",
  { path: "./report.pdf" },
  {
    title: "Annual Report 2024",
    author: "Finance Team",
    year: 2024,
    category: "financials",
    tags: ["annual-report", "investor-relations"],
    categories: ["finance", "public-company"],
  },
);

Metadata values can also include lists of strings or numbers, for example tags or categories.See all supported metadata types here.

Processing documents

After a document is uploaded, TopK returns a processing handle — a reference to the processing job. The document is not ready for retrieval until it has been processed. Processing happens asynchronously in the background and may take a few seconds to a few minutes, depending on document size and complexity.

Your documents might not be ready for retrieval immediately after upload.

Checking the status

After uploading a document, you can check whether it has been processed using the processing handle.

Python SDK
JavaScript SDK

import os
from topk_sdk import Client

client = Client(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

processed = client.dataset("my-docs").check_handle(handle)
print(processed)  # True or False

import { Client } from "topk-js";

const client = new Client({
  apiKey: process.env.TOPK_API_KEY!,
  region: process.env.TOPK_REGION!,
});

const processed = await client.dataset("my-docs").checkHandle(handle);

console.log(processed);  // true or false

Waiting for completion

Alternatively, you can wait for the document to be processed using the processing handle:

CLI
Python SDK
JavaScript SDK

Pass --wait to block until processing is complete:

topk upload ./report.pdf --dataset my-docs --wait

import os
from topk_sdk import Client

client = Client(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

client.dataset("my-docs").wait_for_handle(handle)

# Document is now ready to be queried

You can customize polling behavior by passing a WaitConfig:

from topk_sdk import WaitConfig

client.dataset("my-docs").wait_for_handle(
    handle,
    WaitConfig(frequency_secs=10, timeout_secs=600),
)

Parameter	Default	Description
`frequency_secs`	`5`	How often to poll for the handle status
`timeout_secs`	`300`	Maximum time to wait before timing out

import { Client } from "topk-js";

const client = new Client({
  apiKey: process.env.TOPK_API_KEY!,
  region: process.env.TOPK_REGION!,
});

await client.dataset("my-docs").waitForHandle(handle);

// Document is now ready to be queried

You can customize polling behavior by passing a WaitConfig:

await client.dataset("my-docs").waitForHandle(handle, {
  frequencySecs: 10,
  timeoutSecs: 600,
});

Parameter	Default	Description
`frequencySecs`	`5`	How often to poll for the handle status
`timeoutSecs`	`300`	Maximum time to wait before timing out

Overview

Core Concepts

Management

Uploading documents

Supported formats

Attaching metadata

Processing documents

Checking the status

Waiting for completion

Overview

Core Concepts

Management

Documentation Index

​Uploading documents

​Supported formats

​Attaching metadata

​Processing documents

​Checking the status

​Waiting for completion

Uploading documents

Supported formats

Attaching metadata

Processing documents

Checking the status

Waiting for completion