Skip to main content
Prerequisites
With TopK’s true hybrid search, you can combine multiple retrieval techniques such as:
  • vector search
  • multi-vector search
  • keyword search
  • metadata filtering
— all in a single query.

​
How TopK differs from other “hybrid” search systems

Most databases that offer hybrid search maintain separate vector and keyword indexes. When a query is executed they:
  1. Run two separate queries for both indexes
  2. Collect the top results from each query (e.g. first 100 + 100 candidates)
  3. Use techniques like Reciprocal Rank Fusion (RRF) to merge and rerank these two sets of results


This approach is fundamentally probabilistic - the final top-k results are not guaranteed to be the actual best candidates because some potential candidates might be missed if they don’t appear in either index’s top results. TopK is different. It runs through a single index(vector + keyword), ensuring that our “top 100” results are the actual top 100 - not just a probabilistic approximation: With TopK, you can:
  • Retrieve documents based on multiple embeddings — Multi-vector retrieval
  • Combine semantic similarity(e.g vector search) with keyword search — True Hybrid Retrieval
  • Filter documents by their metadata
  • Apply custom scoring functions blending multiple ranking factors — Custom scoring

​
Implementing Hybrid Search (Vector + Keyword)

Hybrid retrieval combines semantic similarity (vector-based search) with exact keyword matching. This approach ensures that documents with direct keyword matches are considered alongside those that are semantically similar to the query. Let’s define a collection with one keyword_index() and one semantic_index(): 
from topk_sdk.schema import text, keyword_index, semantic_index

client.collections().create(
    "articles",
    schema={
        "title": text().required().index(keyword_index()),  # Keyword-based retrieval
        "content": text().index(semantic_index()),  # Semantic search
    },
)

import { text, keywordIndex, semanticIndex } from "topk-js/schema";

await client.collections().create("articles", {
  title: text().required().index(keywordIndex()), // Keyword-based retrieval
  content: text().index(semanticIndex()), // Semantic search
});

CREATE TABLE articles (
  title   TEXT NOT NULL INDEX keyword_index(),
  content TEXT         INDEX semantic_index()
);
In the following example we’ll perform a hybrid search that combines keyword and vector(semantic) search in a single query: 
from topk_sdk.query import select, field, fn, match

docs = client.collection("articles").query(
    select(
        "title",
        content_similarity=fn.semantic_similarity("content", "climate change policies"),
        text_score=fn.bm25_score(),
    )
    .filter(match("carbon") | match("renewable energy"))  # Ensure keyword relevance
    .sort(field("content_similarity") * 0.6 + field("text_score") * 0.4, asc=False).limit(10)
)

import { select, field, fn, match } from "topk-js/query";

const docs = await client.collection("articles").query(
  select({
    title: field("title"),
    content_similarity: fn.semanticSimilarity(
      "content",
      "climate change policies"
    ),
    text_score: fn.bm25Score(),
  })
    .filter(match("carbon").or(match("renewable energy"))) // Ensure keyword relevance
    .sort(
      field("content_similarity").mul(0.6).add(field("text_score").mul(0.4)),
      false
    ).limit(10)
);

SELECT
  title,
  semantic_similarity(content, 'climate change policies') AS content_similarity,
  bm25_score() AS text_score
FROM articles
WHERE match('carbon', title) OR match('renewable energy', title)
ORDER BY content_similarity * 0.6 + text_score * 0.4 DESC
LIMIT 10;
Let’s break down the example above:
  • We retrieve documents based on semantic meaning (content_similarity) and keyword matching (text_score).
  • The filter() ensures that documents contain at least one relevant keyword.
  • The topk() function weights the scores, prioritizing semantic meaning (60%) while still considering keyword matches (40%).
This balances precision and recall, capturing both exact keyword matches and meaningful context.

​
Implementing Complex Search(Keyword + Vector + Filtering)

In TopK, you can combine keyword search, vector search, and filtering in a single query. This allows you to fetch the truly most relevant results while maintaining a steady performance - no overfetching. 
from topk_sdk.query import select, field, fn, match

docs = client.collection("books").query(
    select(
        "title",
        # Score documents using BM25 algorithm
        text_score=fn.bm25_score(),
        # Compute semantic similarity between the provided query and the `title` field.
        title_similarity=fn.semantic_similarity("title", "catcher"),
    )
    # Filter documents that contain the `great` keyword
    .filter(match("great"))
    # Filtering by metadata
    .filter(field("published_year") > 1980)
    # Return top 10 documents with the highest combined score
    .sort(field("text_score") * 0.2 + field("title_similarity") * 0.8, asc=False).limit(10)
)

import { select, fn, field, match } from "topk-js/query";

const docs = await client.collection("books").query(
  select({
    title: field("title"),
    text_score: fn.bm25Score(),
    title_similarity: fn.semanticSimilarity("title", "catcher"),
  })
    .filter(match("great"))
    .filter(field("published_year").gt(1980))
    .sort(
      field("text_score").mul(0.2).add(field("title_similarity").mul(0.8)),
      false
    ).limit(10)
);

-- In SQL, keyword and semantic indexes must be on separate fields (unlike the SDK).
-- This uses 'articles' from above: title (keyword_index) + content (semantic_index).
SELECT
  title,
  semantic_similarity(content, 'climate policy') AS content_similarity
FROM articles
WHERE match_any(title, 'carbon')
  AND published_year > 2020
ORDER BY content_similarity DESC
LIMIT 10;
As you might have noticed, we are also sorting the top-k results using a custom scoring function. You can read more about custom scoring functions in the following section.

​
Custom Scoring Functions

TopK allows you to define custom scoring functions by combining:
  • Semantic similarity score
  • Keyword score(BM25)
  • Vector distance
  • “Bring-your-own” precomputed importance score

​
Defining a Collection with Custom Scoring Fields

from topk_sdk.schema import text, float, semantic_index

client.collections().create(
    "documents",
    schema={
        "content": text().index(semantic_index()),  # Semantic search
        "importance": float().required(),  # Precomputed importance score
    },
)

import { text, float, semanticIndex } from "topk-js/schema";

await client.collections().create("documents", {
  content: text().index(semanticIndex()), // Semantic search
  importance: float().required(), // Precomputed importance score
});

CREATE TABLE documents (
  content    TEXT  INDEX semantic_index(),
  importance FLOAT NOT NULL
);

​
Querying with a Custom Scoring Function

from topk_sdk.query import select, field, fn

docs = client.collection("documents").query(
    select(
        "content",
        "importance",
        content_score=fn.semantic_similarity("content", "machine learning applications"),
    )
    .sort(0.8 * field("importance") + 0.2 * field("content_score"), asc=False).limit(10)
)

import { select, field, fn } from "topk-js/query";

const docs = await client.collection("documents").query(
  select({
    content: field("content"),
    importance: field("importance"),
    content_score: fn.semanticSimilarity(
      "content",
      "machine learning applications"
    ),
  }).sort(
    field("importance").mul(0.8).add(field("content_score").mul(0.2)),
    false
  ).limit(10)
);

SELECT
  content,
  importance,
  semantic_similarity(content, 'machine learning applications') AS content_score
FROM documents
ORDER BY importance * 0.8 + content_score * 0.2 DESC
LIMIT 10;
Let’s break down the example above:
  1. First, we retrieve documents based on both semantic similarity (content_score) and precomputed importance (importance_score).
  2. Then, the topk() function gives 80% weight to content score and 20% weight to document importance.
  3. Sorting by a custom scoring function allows us to boost more critical documents, ensuring that highly relevant but less “important” content doesn’t dominate.