Ingest Files

To run queries in TopK, your documents need to be uploaded and processed. Once a document is uploaded, TopK will process it to create searchable chunks with relevant context and metadata. This indexed representation then powers Ask and Search.

Uploading documents

Documents are uploaded to a dataset. A dataset is a named container for your documents. Each document in a dataset is identified by a document ID.

Learn how to create your dataset here.

CLI
Python SDK
JavaScript SDK

# Upload all supported files in the directory
topk upload ./docs --dataset my-docs

# Upload all supported files in the directory and subdirectories recursively
topk upload ./docs -r --dataset my-docs

# Upload all supported files in the directory by glob pattern
topk upload "./docs/*" --dataset my-docs

# Upload all supported files in the directory by glob pattern recursively
topk upload "./docs/**/*" -r --dataset my-docs

# Upload all PDFs in the directory by glob pattern
topk upload "./docs/*.pdf" --dataset my-docs

# Upload all PDFs in the directory by glob pattern recursively
topk upload "./docs/**/*.pdf" --dataset my-docs

# Upload a single file
topk upload ./report.pdf --dataset my-docs

import os
from pathlib import Path
from topk_sdk import Client

client = Client(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

handle = client.dataset("my-docs").upsert_file(
    "report-2024",                       # document ID
    Path("./report.pdf"),                # path to file
    {"title": "Annual Report 2024"},     # document metadata (optional)
)

print(handle)  # e.g. "hdl_abc123"

import os
from pathlib import Path
from topk_sdk import AsyncClient

client = AsyncClient(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

handle = await client.dataset("my-docs").upsert_file(
    "report-2024",                       # document ID
    Path("./report.pdf"),                # path to file
    {"title": "Annual Report 2024"},     # document metadata (optional)
)

print(handle)  # e.g. "hdl_abc123"

import { Client } from "topk-js";

const client = new Client({
  apiKey: process.env.TOPK_API_KEY!,
  region: process.env.TOPK_REGION!,
});

const handle = await client.dataset("my-docs").upsertFile(
  "report-2024",                    // document ID
  { path: "./report.pdf" },         // path to file
  { title: "Annual Report 2024" },  // document metadata (optional)
);

console.log(handle);  // e.g. "hdl_abc123"

Supported formats

Format	MIME type	Extensions
PDF	`application/pdf`	`.pdf`
Markdown	`text/markdown`	`.md`
HTML	`text/html`	`.html`
PNG	`image/png`	`.png`
JPEG	`image/jpeg`	`.jpg`, `.jpeg`
GIF	`image/gif`	`.gif`
WebP	`image/webp`	`.webp`
TIFF	`image/tiff`	`.tiff`, `.tif`
BMP	`image/bmp`	`.bmp`

Attaching metadata

Each document can be associated with its own metadata. You can attach additional information about the document, such as title, author, date, or category.

Python SDK
JavaScript SDK

from pathlib import Path

resp = client.dataset("my-docs").upsert_file(
    "report-2024",
    Path("./report.pdf"),
    {
        "title": "Annual Report 2024",
        "author": "Finance Team",
        "year": 2024,
        "category": "financials",
        "tags": ["annual-report", "investor-relations"],
        "categories": ["finance", "public-company"],
    },
)

from pathlib import Path

resp = await client.dataset("my-docs").upsert_file(
    "report-2024",
    Path("./report.pdf"),
    {
        "title": "Annual Report 2024",
        "author": "Finance Team",
        "year": 2024,
        "category": "financials",
        "tags": ["annual-report", "investor-relations"],
        "categories": ["finance", "public-company"],
    },
)

const resp = await client.dataset("my-docs").upsertFile(
  "report-2024",
  { path: "./report.pdf" },
  {
    title: "Annual Report 2024",
    author: "Finance Team",
    year: 2024,
    category: "financials",
    tags: ["annual-report", "investor-relations"],
    categories: ["finance", "public-company"],
  },
);

Metadata values can also include lists of strings or numbers, for example tags or categories.See all supported metadata types here.

Processing documents

After a document is uploaded, TopK returns a processing handle — a reference to the processing job. The document is not ready for retrieval until it has been processed. Processing happens asynchronously in the background and may take a few seconds to a few minutes, depending on document size and complexity.

Your documents might not be ready for retrieval immediately after upload.

Checking the status

After uploading a document, you can check whether it has been processed using the processing handle.

Python SDK
JavaScript SDK

import os
from topk_sdk import Client

client = Client(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

processed = client.dataset("my-docs").check_handle(handle)
print(processed)  # True or False

import os
from topk_sdk import AsyncClient

client = AsyncClient(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

processed = await client.dataset("my-docs").check_handle(handle)

print(processed)  # True or False

import { Client } from "topk-js";

const client = new Client({
  apiKey: process.env.TOPK_API_KEY!,
  region: process.env.TOPK_REGION!,
});

const processed = await client.dataset("my-docs").checkHandle(handle);

console.log(processed);  // true or false

Waiting for completion

Alternatively, you can wait for the document to be processed using the processing handle:

CLI
Python SDK
JavaScript SDK

Pass --wait to block until processing is complete:

topk upload ./report.pdf --dataset my-docs --wait

import os
from topk_sdk import Client

client = Client(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

client.dataset("my-docs").wait_for_handle(handle)

# Document is now ready to be queried

import os
from topk_sdk import AsyncClient

client = AsyncClient(
    api_key=os.environ["TOPK_API_KEY"],
    region=os.environ["TOPK_REGION"],
)

await client.dataset("my-docs").wait_for_handle(handle)

# Document is now ready to be queried

You can customize polling behavior by passing a WaitConfig:

from topk_sdk import WaitConfig

client.dataset("my-docs").wait_for_handle(
    handle,
    WaitConfig(frequency_secs=10, timeout_secs=600),
)

from topk_sdk import WaitConfig

await client.dataset("my-docs").wait_for_handle(
    handle,
    WaitConfig(frequency_secs=10, timeout_secs=600),
)

Parameter	Default	Description
`frequency_secs`	`5`	How often to poll for the handle status
`timeout_secs`	`300`	Maximum time to wait before timing out

import { Client } from "topk-js";

const client = new Client({
  apiKey: process.env.TOPK_API_KEY!,
  region: process.env.TOPK_REGION!,
});

await client.dataset("my-docs").waitForHandle(handle);

// Document is now ready to be queried

You can customize polling behavior by passing a WaitConfig:

await client.dataset("my-docs").waitForHandle(handle, {
  frequencySecs: 10,
  timeoutSecs: 600,
});

Parameter	Default	Description
`frequencySecs`	`5`	How often to poll for the handle status
`timeoutSecs`	`300`	Maximum time to wait before timing out

Overview

Dataset API

Collection API

Management API

Ingest Files

Uploading documents

Supported formats

Attaching metadata

Processing documents

Checking the status

Waiting for completion

Next Steps

Ask Questions

Search Chunks

​Uploading documents

​Supported formats

​Attaching metadata

​Processing documents

​Checking the status

​Waiting for completion

​Next Steps

Ask Questions

Search Chunks

Uploading documents

Supported formats

Attaching metadata

Processing documents

Checking the status

Waiting for completion

Next Steps