Metadata filtering

In addition to providing an input vector to your query, you can also filter by vector metadata associated with every vector. Query results will only include vectors that match the filter criteria, meaning that filter is applied first, and the topK results are taken from the filtered set.

By using metadata filtering to limit the scope of a query, you can filter by specific customer IDs, tenant, product category or any other metadata you associate with your vectors.

Metadata indexes

Vectorize supports namespace filtering by default, but to filter on another metadata property of your vectors, you’ll need to create a metadata index. You can create up to 10 metadata indexes per Vectorize index.

Metadata indexes for properties of type string, number and boolean are supported. Please refer to Create metadata indexes for details.

You can store up to 10KiB of metadata per vector. See Vectorize Limits for a complete list of limits.

For metadata indexes of type number, the indexed number precision is that of float64.

For metadata indexes of type string, each vector indexes the first 64B of the string data truncated on UTF-8 character boundaries to the longest well-formed UTF-8 substring within that limit, so vectors are filterable on the first 64B of their value for each indexed property.

Enable metadata filtering

Vectors upserted before a metadata index was created won’t have their metadata contained in that index. Upserting/re-upserting vectors after it was created will have them indexed as expected. Please refer to Create metadata indexes for details.

Supported operations

Optional filter property on query() method specifies metadata filter:

Operator	Description
$eq	Equals
$ne	Not equals

• filter must be non-empty object whose compact JSON representation must be less than 2048 bytes.
• filter object keys cannot be empty, contain " | . (dot is reserved for nesting), start with $, or be longer than 512 characters.
• filter object non-nested values can be string, number, boolean, or null values.

Namespace versus metadata filtering

Both namespaces and metadata filtering narrow the vector search space for a query. Consider the following when evaluating both filter types:

• A namespace filter is applied before metadata filter(s).
• A vector can only be part of a single namespace with the documented limits. Vector metadata can contain multiple key-value pairs up to metadata per vector limits. Metadata values support different types (string, boolean, and others), therefore offering more flexibility.

Valid filter examples

Implicit $eq operator

{ "streaming_platform": "netflix" }

Explicit operator

{ "someKey": { "$ne": "hbo" } }

Implicit logical AND with multiple keys

{ "pandas.nice": 42, "someKey": { "$ne": "someValue" } }

Keys define nesting with . (dot)

{ "pandas.nice": 42 }


// looks for { "pandas": { "nice": 42 } }

Examples

Add metadata

Using legacy Vectorize (V1) indexes?

Please use the wrangler vectorize --deprecated-v1 flag to create, get, list, delete and insert vectors into legacy Vectorize V1 indexes.

Please note that by December 2024, you will not be able to create legacy Vectorize indexes. Other operations will remain functional.

Refer to the legacy transition page for more details on transitioning away from legacy indexes.

With the following index definition:

npx wrangler vectorize create tutorial-index --dimensions=32 --metric=cosine

Create metadata indexes:

npx wrangler vectorize create-metadata-index tutorial-index --property-name=url --type=string

npx wrangler vectorize create-metadata-index tutorial-index --property-name=streaming_platform --type=string

Metadata can be added when inserting or upserting vectors.

const newMetadataVectors: Array<VectorizeVector> = [
  {
    id: "1",
    values: [32.4, 74.1, 3.2, ...],
    metadata: { url: "/products/sku/13913913", streaming_platform: "netflix" },
  },
  {
    id: "2",
    values: [15.1, 19.2, 15.8, ...],
    metadata: { url: "/products/sku/10148191", streaming_platform: "hbo" },
  },
  {
    id: "3",
    values: [0.16, 1.2, 3.8, ...],
    metadata: { url: "/products/sku/97913813", streaming_platform: "amazon" },
  },
  {
    id: "4",
    values: [75.1, 67.1, 29.9, ...],
    metadata: { url: "/products/sku/418313", streaming_platform: "netflix" },
  },
  {
    id: "5",
    values: [58.8, 6.7, 3.4, ...],
    metadata: { url: "/products/sku/55519183", streaming_platform: "hbo" },
  },
];

// Upsert vectors with added metadata, returning a count of the vectors upserted and their vector IDs
let upserted = await env.YOUR_INDEX.upsert(newMetadataVectors);

Query examples

Use the query() method:

let queryVector: Array<number> = [54.8, 5.5, 3.1, ...];
let originalMatches = await env.YOUR_INDEX.query(queryVector, {
  topK: 3,
  returnValues: true,
  returnMetadata: 'all',
});

Results without metadata filtering:

{
  "count": 3,
  "matches": [
    {
      "id": "5",
      "score": 0.999909486,
      "values": [58.79999923706055, 6.699999809265137, 3.4000000953674316],
      "metadata": {
        "url": "/products/sku/55519183",
        "streaming_platform": "hbo"
      }
    },
    {
      "id": "4",
      "score": 0.789848214,
      "values": [75.0999984741211, 67.0999984741211, 29.899999618530273],
      "metadata": {
        "url": "/products/sku/418313",
        "streaming_platform": "netflix"
      }
    },
    {
      "id": "2",
      "score": 0.611976262,
      "values": [15.100000381469727, 19.200000762939453, 15.800000190734863],
      "metadata": {
        "url": "/products/sku/10148191",
        "streaming_platform": "hbo"
      }
    }
  ]
}

The same query() method with a filter property supports metadata filtering.

let queryVector: Array<number> = [54.8, 5.5, 3.1, ...];
let metadataMatches = await env.YOUR_INDEX.query(queryVector, {
  topK: 3,
  filter: { streaming_platform: "netflix" },
  returnValues: true,
  returnMetadata: 'all',
});

Results with metadata filtering:

{
  "count": 2,
  "matches": [
    {
      "id": "4",
      "score": 0.789848214,
      "values": [75.0999984741211, 67.0999984741211, 29.899999618530273],
      "metadata": {
        "url": "/products/sku/418313",
        "streaming_platform": "netflix"
      }
    },
    {
      "id": "1",
      "score": 0.491185264,
      "values": [32.400001525878906, 74.0999984741211, 3.200000047683716],
      "metadata": {
        "url": "/products/sku/13913913",
        "streaming_platform": "netflix"
      }
    }
  ]
}

Limitations

• As of now, metadata indexes need to be created for Vectorize indexes before vectors can be inserted to support metadata filtering.
• Only indexes created on or after 2023-12-06 support metadata filtering. Previously created indexes cannot be migrated to support metadata filtering.