Namespace faiss

namespace faiss

Implementation of k-means clustering with many variants.

Copyright (c) Facebook, Inc. and its affiliates.

This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.

IDSelector is intended to define a subset of vectors to handle (for removal or as subset to search)

PQ4 SIMD packing and accumulation functions

The basic kernel accumulates nq query vectors with bbs = nb * 2 * 16 vectors and produces an output matrix for that. It is interesting for nq * nb <= 4, otherwise register spilling becomes too large.

The implementation of these functions is spread over 3 cpp files to reduce parallel compile times. Templates are instantiated explicitly.

This file contains callbacks for kernels that compute distances.

Throughout the library, vectors are provided as float * pointers. Most algorithms can be optimized when several vectors are processed (added/searched) together in a batch. In this case, they are passed in as a matrix. When n vectors of size d are provided as float * x, component j of vector i is

x[ i * d + j ]

where 0 <= i < n and 0 <= j < d. In other words, matrices are always compact. When specifying the size of the matrix, we call it an n*d matrix, which implies a row-major storage.

I/O functions can read/write to a filename, a file handle or to an object that abstracts the medium.

The read functions return objects that should be deallocated with delete. All references within these objectes are owned by the object.

Definition of inverted lists + a few common classes that implement the interface.

Since IVF (inverted file) indexes are of so much use for large-scale use cases, we group a few functions related to them in this small library. Most functions work both on IndexIVFs and IndexIVFs embedded within an IndexPreTransform.

In this file are the implementations of extra metrics beyond L2 and inner product

Implements a few neural net layers, mainly to support QINCo

Defines a few objects that apply transformations to a set of vectors Often these are pre-processing steps.

Typedefs

using IndexIDMap = IndexIDMapTemplate<Index>

using IndexBinaryIDMap = IndexIDMapTemplate<IndexBinary>

using IndexIDMap2 = IndexIDMap2Template<Index>

using IndexBinaryIDMap2 = IndexIDMap2Template<IndexBinary>

using IVFSearchParameters = SearchParametersIVF

using IndexReplicas = IndexReplicasTemplate<Index>

using IndexBinaryReplicas = IndexReplicasTemplate<IndexBinary>

using IndexShards = IndexShardsTemplate<Index>

using IndexBinaryShards = IndexShardsTemplate<IndexBinary>

using ConcatenatedInvertedLists = HStackInvertedLists

using idx_t = int64_t

all vector indices are this type

typedef HeapArray<CMin<float, int64_t>> float_minheap_array_t

typedef HeapArray<CMin<int, int64_t>> int_minheap_array_t

typedef HeapArray<CMax<float, int64_t>> float_maxheap_array_t

typedef HeapArray<CMax<int, int64_t>> int_maxheap_array_t

Enums

enum MetricType

The metric space for vector comparison for Faiss indices and algorithms.

Most algorithms support both inner product and L2, with the flat (brute-force) indices supporting additional metric types for vector comparison.

Values:

enumerator METRIC_INNER_PRODUCT

maximum inner product search

enumerator METRIC_L2

squared L2 search

enumerator METRIC_L1

L1 (aka cityblock)

enumerator METRIC_Linf

infinity distance

enumerator METRIC_Lp

L_p distance, p is given by a faiss::Index metric_arg

enumerator METRIC_Canberra

some additional metrics defined in scipy.spatial.distance

enumerator METRIC_BrayCurtis

enumerator METRIC_JensenShannon

enumerator METRIC_Jaccard

sum_i(min(a_i, b_i)) / sum_i(max(a_i, b_i)) where a_i, b_i > 0

enumerator METRIC_NaNEuclidean

Squared Eucliden distance, ignoring NaNs.

enumerator METRIC_ABS_INNER_PRODUCT

abs(x | y): the distance to a hyperplane

Functions

Index *clone_index(const Index*)

Quantizer *clone_Quantizer(const Quantizer *quant)

IndexBinary *clone_binary_index(const IndexBinary *index)

float kmeans_clustering(size_t d, size_t n, size_t k, const float *x, float *centroids)

simplified interface

Parameters:

• d – dimension of the data

• n – nb of training vectors

• k – nb of output centroids

• x – training set (size n * d)

• centroids – output centroids (size k * d)

Returns:

final quantization error

std::string reverse_index_factory(const faiss::Index *index)

std::string reverse_index_factory(const faiss::IndexBinary *index)

template<typename PQDecoderT>
inline float distance_single_code_generic(const size_t M, const size_t nbits, const float *sim_table, const uint8_t *code)

Returns the distance to a single code.

template<typename PQDecoderT> inline void distance_four_codes_generic (const size_t M, const size_t nbits, const float *sim_table, const uint8_t *__restrict code0, const uint8_t *__restrict code1, const uint8_t *__restrict code2, const uint8_t *__restrict code3, float &result0, float &result1, float &result2, float &result3)

Combines 4 operations of distance_single_code() General-purpose version.

template<typename PQDecoderT>
inline float distance_single_code(const size_t M, const size_t nbits, const float *sim_table, const uint8_t *code)

template<typename PQDecoderT> inline void distance_four_codes (const size_t M, const size_t nbits, const float *sim_table, const uint8_t *__restrict code0, const uint8_t *__restrict code1, const uint8_t *__restrict code2, const uint8_t *__restrict code3, float &result0, float &result1, float &result2, float &result3)

void handleExceptions(std::vector<std::pair<int, std::exception_ptr>> &exceptions)

Handle multiple exceptions from worker threads, throwing an appropriate exception that aggregates the information The pair int is the thread that generated the exception

std::string demangle_cpp_symbol(const char *name)

make typeids more readable

int search_from_candidates(const HNSW &hnsw, DistanceComputer &qdis, ResultHandler<HNSW::C> &res, HNSW::MinimaxHeap &candidates, VisitedTable &vt, HNSWStats &stats, int level, int nres_in = 0, const SearchParametersHNSW *params = nullptr)

HNSWStats greedy_update_nearest(const HNSW &hnsw, DistanceComputer &qdis, int level, HNSW::storage_idx_t &nearest, float &d_nearest)

std::priority_queue<HNSW::Node> search_from_candidate_unbounded(const HNSW &hnsw, const HNSW::Node &node, DistanceComputer &qdis, int ef, VisitedTable *vt, HNSWStats &stats)

void search_neighbors_to_add(HNSW &hnsw, DistanceComputer &qdis, std::priority_queue<HNSW::NodeDistCloser> &results, int entry_point, float d_entry_point, int level, VisitedTable &vt, bool reference_version = false)

void read_index_header(Index *idx, IOReader *f)

void read_direct_map(DirectMap *dm, IOReader *f)

void read_ivf_header(IndexIVF *ivf, IOReader *f, std::vector<std::vector<idx_t>> *ids = nullptr)

void read_InvertedLists(IndexIVF *ivf, IOReader *f, int io_flags)

ArrayInvertedLists *set_array_invlist(IndexIVF *ivf, std::vector<std::vector<idx_t>> &ids)

void read_ProductQuantizer(ProductQuantizer *pq, IOReader *f)

void read_ScalarQuantizer(ScalarQuantizer *ivsc, IOReader *f)

uint32_t fourcc(const char sx[4])

cast a 4-character string to a uint32_t that can be written and read easily

uint32_t fourcc(const std::string &sx)

void fourcc_inv(uint32_t x, char str[5])

std::string fourcc_inv(uint32_t x)

std::string fourcc_inv_printable(uint32_t x)

void smawk(const idx_t nrows, const idx_t ncols, const float *x, idx_t *argmins)

SMAWK algorithm. Find the row minima of a monotone matrix.

Expose this for testing.

Parameters:

• nrows – number of rows

• ncols – number of columns

• x – input matrix, size (nrows, ncols)

• argmins – argmin of each row

double kmeans1d(const float *x, size_t n, size_t nclusters, float *centroids)

Exact 1D K-Means by dynamic programming

From “Fast Exact k-Means, k-Medians and Bregman Divergence Clustering in 1D” Allan Grønlund, Kasper Green Larsen, Alexander Mathiasen, Jesper Sindahl Nielsen, Stefan Schneider, Mingzhou Song, ArXiV’17

Section 2.2

https://arxiv.org/abs/1701.07204

Parameters:

• x – input 1D array

• n – input array length

• nclusters – number of clusters

• centroids – output centroids, size nclusters

Returns:

imbalancce factor

void pq4_pack_codes(const uint8_t *codes, size_t ntotal, size_t M, size_t nb, size_t bbs, size_t nsq, uint8_t *blocks)

Pack codes for consumption by the SIMD kernels. The unused bytes are set to 0.

Parameters:

• codes – input codes, size (ntotal, ceil(M / 2))

• ntotal – number of input codes

• nb – output number of codes (ntotal rounded up to a multiple of bbs)

• nsq – number of sub-quantizers (=M rounded up to a muliple of 2)

• bbs – size of database blocks (multiple of 32)

• blocks – output array, size nb * nsq / 2.

void pq4_pack_codes_range(const uint8_t *codes, size_t M, size_t i0, size_t i1, size_t bbs, size_t nsq, uint8_t *blocks)

Same as pack_codes but write in a given range of the output, leaving the rest untouched. Assumes allocated entries are 0 on input.

Parameters:

• codes – input codes, size (i1 - i0, ceil(M / 2))

• i0 – first output code to write

• i1 – last output code to write

• blocks – output array, size at least ceil(i1 / bbs) * bbs * nsq / 2

uint8_t pq4_get_packed_element(const uint8_t *data, size_t bbs, size_t nsq, size_t vector_id, size_t sq)

get a single element from a packed codes table

Parameters:

• vector_id – vector id

• sq – subquantizer (< nsq)

void pq4_set_packed_element(uint8_t *data, uint8_t code, size_t bbs, size_t nsq, size_t vector_id, size_t sq)

set a single element “code” into a packed codes table

Parameters:

• vector_id – vector id

• sq – subquantizer (< nsq)

void pq4_pack_LUT(int nq, int nsq, const uint8_t *src, uint8_t *dest)

Pack Look-up table for consumption by the kernel.

Parameters:

• nq – number of queries

• nsq – number of sub-quantizers (muliple of 2)

• src – input array, size (nq, 16)

• dest – output array, size (nq, 16)

void pq4_accumulate_loop(int nq, size_t nb, int bbs, int nsq, const uint8_t *codes, const uint8_t *LUT, SIMDResultHandler &res, const NormTableScaler *scaler)

Loop over database elements and accumulate results into result handler

Parameters:

• nq – number of queries

• nb – number of database elements

• bbs – size of database blocks (multiple of 32)

• nsq – number of sub-quantizers (muliple of 2)

• codes – packed codes array

• LUT – packed look-up table

• scaler – scaler to scale the encoded norm

int pq4_qbs_to_nq(int qbs)

int pq4_preferred_qbs(int nq)

return the preferred decomposition in blocks for a nb of queries.

int pq4_pack_LUT_qbs(int fqbs, int nsq, const uint8_t *src, uint8_t *dest)

Pack Look-up table for consumption by the kernel.

Parameters:

• qbs – 4-bit encoded number of query blocks, the total number of queries handled (nq) is deduced from it

• nsq – number of sub-quantizers (muliple of 2)

• src – input array, size (nq, 16)

• dest – output array, size (nq, 16)

Returns:

nq

int pq4_pack_LUT_qbs_q_map(int qbs, int nsq, const uint8_t *src, const int *q_map, uint8_t *dest)

Same as pq4_pack_LUT_qbs, except the source vectors are remapped with q_map

void pq4_accumulate_loop_qbs(int qbs, size_t nb, int nsq, const uint8_t *codes, const uint8_t *LUT, SIMDResultHandler &res, const NormTableScaler *scaler = nullptr)

Run accumulation loop.

Parameters:

• qbs – 4-bit encoded number of queries

• nb – number of database codes (mutliple of bbs)

• nsq – number of sub-quantizers

• codes – encoded database vectors (packed)

• LUT – look-up table (packed)

• res – call-back for the resutls

• scaler – scaler to scale the encoded norm

void beam_search_encode_step(size_t d, size_t K, const float *cent, size_t n, size_t beam_size, const float *residuals, size_t m, const int32_t *codes, size_t new_beam_size, int32_t *new_codes, float *new_residuals, float *new_distances, Index *assign_index = nullptr, ApproxTopK_mode_t approx_topk = ApproxTopK_mode_t::EXACT_TOPK)

Encode a residual by sampling from a centroid table.

This is a single encoding step the residual quantizer. It allows low-level access to the encoding function, exposed mainly for unit tests.

Parameters:

• n – number of vectors to handle

• residuals – vectors to encode, size (n, beam_size, d)

• cent – centroids, size (K, d)

• beam_size – input beam size

• m – size of the codes for the previous encoding steps

• codes – code array for the previous steps of the beam (n, beam_size, m)

• new_beam_size – output beam size (should be <= K * beam_size)

• new_codes – output codes, size (n, new_beam_size, m + 1)

• new_residuals – output residuals, size (n, new_beam_size, d)

• new_distances – output distances, size (n, new_beam_size)

• assign_index – if non-NULL, will be used to perform assignment

void beam_search_encode_step_tab(size_t K, size_t n, size_t beam_size, const float *codebook_cross_norms, size_t ldc, const uint64_t *codebook_offsets, const float *query_cp, size_t ldqc, const float *cent_norms_i, size_t m, const int32_t *codes, const float *distances, size_t new_beam_size, int32_t *new_codes, float *new_distances, ApproxTopK_mode_t approx_topk = ApproxTopK_mode_t::EXACT_TOPK)

Encode a set of vectors using their dot products with the codebooks

Parameters:

• K – number of vectors in the codebook

• n – nb of vectors to encode

• beam_size – input beam size

• codebook_cross_norms – inner product of this codebook with the m previously encoded codebooks

• codebook_offsets – offsets into codebook_cross_norms for each previous codebook

• query_cp – dot products of query vectors with ???

• cent_norms_i – norms of centroids

template<class Consumer, class ...Types>
Consumer::T dispatch_knn_ResultHandler(size_t nx, float *vals, int64_t *ids, size_t k, MetricType metric, const IDSelector *sel, Consumer &consumer, Types... args)

template<class Consumer, class ...Types>
Consumer::T dispatch_range_ResultHandler(RangeSearchResult *res, float radius, MetricType metric, const IDSelector *sel, Consumer &consumer, Types... args)

Index *index_factory(int d, const char *description, MetricType metric = METRIC_L2)

Build and index with the sequence of processing steps described in the string.

IndexBinary *index_binary_factory(int d, const char *description)

void write_index(const Index *idx, const char *fname, int io_flags = 0)

void write_index(const Index *idx, FILE *f, int io_flags = 0)

void write_index(const Index *idx, IOWriter *writer, int io_flags = 0)

void write_index_binary(const IndexBinary *idx, const char *fname)

void write_index_binary(const IndexBinary *idx, FILE *f)

void write_index_binary(const IndexBinary *idx, IOWriter *writer)

Index *read_index(const char *fname, int io_flags = 0)

Index *read_index(FILE *f, int io_flags = 0)

Index *read_index(IOReader *reader, int io_flags = 0)

IndexBinary *read_index_binary(const char *fname, int io_flags = 0)

IndexBinary *read_index_binary(FILE *f, int io_flags = 0)

IndexBinary *read_index_binary(IOReader *reader, int io_flags = 0)

void write_VectorTransform(const VectorTransform *vt, const char *fname)

void write_VectorTransform(const VectorTransform *vt, IOWriter *f)

VectorTransform *read_VectorTransform(const char *fname)

VectorTransform *read_VectorTransform(IOReader *f)

ProductQuantizer *read_ProductQuantizer(const char *fname)

ProductQuantizer *read_ProductQuantizer(IOReader *reader)

void write_ProductQuantizer(const ProductQuantizer *pq, const char *fname)

void write_ProductQuantizer(const ProductQuantizer *pq, IOWriter *f)

void write_InvertedLists(const InvertedLists *ils, IOWriter *f)

InvertedLists *read_InvertedLists(IOReader *reader, int io_flags = 0)

void initialize_IVFPQ_precomputed_table(int &use_precomputed_table, const Index *quantizer, const ProductQuantizer &pq, AlignedTable<float> &precomputed_table, bool by_residual, bool verbose)

Pre-compute distance tables for IVFPQ with by-residual and METRIC_L2

Parameters:

• use_precomputed_table – (I/O) =-1: force disable =0: decide heuristically (default: use tables only if they are < precomputed_tables_max_bytes), set use_precomputed_table on output =1: tables that work for all quantizers (size 256 * nlist * M) =2: specific version for MultiIndexQuantizer (much more compact)

• precomputed_table – precomputed table to initialize

inline uint64_t lo_build(uint64_t list_id, uint64_t offset)

inline uint64_t lo_listno(uint64_t lo)

inline uint64_t lo_offset(uint64_t lo)

constexpr bool is_similarity_metric(MetricType metric_type)

this function is used to distinguish between min and max indexes since we need to support similarity and dis-similarity metrics in a flexible way

template<int A = 32>
inline bool is_aligned_pointer(const void *x)

inline uint16_t encode_bf16(const float f)

inline float decode_bf16(const uint16_t v)

float fvec_L2sqr(const float *x, const float *y, size_t d)

Squared L2 distance between two vectors.

float fvec_inner_product(const float *x, const float *y, size_t d)

inner product

float fvec_L1(const float *x, const float *y, size_t d)

L1 distance.

float fvec_Linf(const float *x, const float *y, size_t d)

infinity distance

void fvec_inner_product_batch_4(const float *x, const float *y0, const float *y1, const float *y2, const float *y3, const size_t d, float &dis0, float &dis1, float &dis2, float &dis3)

Special version of inner product that computes 4 distances between x and yi, which is performance oriented.

void fvec_L2sqr_batch_4(const float *x, const float *y0, const float *y1, const float *y2, const float *y3, const size_t d, float &dis0, float &dis1, float &dis2, float &dis3)

Special version of L2sqr that computes 4 distances between x and yi, which is performance oriented.

void pairwise_L2sqr(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)

Compute pairwise distances between sets of vectors

Parameters:

• d – dimension of the vectors

• nq – nb of query vectors

• nb – nb of database vectors

• xq – query vectors (size nq * d)

• xb – database vectors (size nb * d)

• dis – output distances (size nq * nb)

• ldq, ldb, ldd – strides for the matrices

void fvec_inner_products_ny(float *ip, const float *x, const float *y, size_t d, size_t ny)

void fvec_L2sqr_ny(float *dis, const float *x, const float *y, size_t d, size_t ny)

void fvec_L2sqr_ny_transposed(float *dis, const float *x, const float *y, const float *y_sqlen, size_t d, size_t d_offset, size_t ny)

size_t fvec_L2sqr_ny_nearest(float *distances_tmp_buffer, const float *x, const float *y, size_t d, size_t ny)

size_t fvec_L2sqr_ny_nearest_y_transposed(float *distances_tmp_buffer, const float *x, const float *y, const float *y_sqlen, size_t d, size_t d_offset, size_t ny)

float fvec_norm_L2sqr(const float *x, size_t d)

squared norm of a vector

void fvec_norms_L2(float *norms, const float *x, size_t d, size_t nx)

compute the L2 norms for a set of vectors

Parameters:

• norms – output norms, size nx

• x – set of vectors, size nx * d

void fvec_norms_L2sqr(float *norms, const float *x, size_t d, size_t nx)

same as fvec_norms_L2, but computes squared norms

void fvec_renorm_L2(size_t d, size_t nx, float *x)

void inner_product_to_L2sqr(float *dis, const float *nr1, const float *nr2, size_t n1, size_t n2)

void fvec_add(size_t d, const float *a, const float *b, float *c)

compute c := a + b for vectors

c and a can overlap, c and b can overlap

Parameters:

• a – size d

• b – size d

• c – size d

void fvec_add(size_t d, const float *a, float b, float *c)

compute c := a + b for a, c vectors and b a scalar

c and a can overlap

Parameters:

• a – size d

• c – size d

void fvec_sub(size_t d, const float *a, const float *b, float *c)

compute c := a - b for vectors

c and a can overlap, c and b can overlap

Parameters:

• a – size d

• b – size d

• c – size d

void fvec_inner_products_by_idx(float *ip, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)

compute the inner product between x and a subset y of ny vectors defined by ids

ip(i, j) = inner_product(x(i, :), y(ids(i, j), :))

Parameters:

• ip – output array, size nx * ny

• x – first-term vector, size nx * d

• y – second-term vector, size (max(ids) + 1) * d

• ids – ids to sample from y, size nx * ny

void fvec_L2sqr_by_idx(float *dis, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)

compute the squared L2 distances between x and a subset y of ny vectors defined by ids

dis(i, j) = inner_product(x(i, :), y(ids(i, j), :))

Parameters:

• dis – output array, size nx * ny

• x – first-term vector, size nx * d

• y – second-term vector, size (max(ids) + 1) * d

• ids – ids to sample from y, size nx * ny

void pairwise_indexed_L2sqr(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)

compute dis[j] = L2sqr(x[ix[j]], y[iy[j]]) forall j=0..n-1

Parameters:

• x – size (max(ix) + 1, d)

• y – size (max(iy) + 1, d)

• ix – size n

• iy – size n

• dis – size n

void pairwise_indexed_inner_product(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)

compute dis[j] = inner_product(x[ix[j]], y[iy[j]]) forall j=0..n-1

Parameters:

• x – size (max(ix) + 1, d)

• y – size (max(iy) + 1, d)

• ix – size n

• iy – size n

• dis – size n

void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_minheap_array_t *res, const IDSelector *sel = nullptr)

Return the k nearest neighbors of each of the nx vectors x among the ny vector y, w.r.t to max inner product.

Parameters:

• x – query vectors, size nx * d

• y – database vectors, size ny * d

• res – result heap structure, which also provides k. Sorted on output

void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes, const IDSelector *sel = nullptr)

Return the k nearest neighbors of each of the nx vectors x among the ny vector y, for the inner product metric.

Parameters:

• x – query vectors, size nx * d

• y – database vectors, size ny * d

• distances – output distances, size nq * k

• indexes – output vector ids, size nq * k

void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_maxheap_array_t *res, const float *y_norm2 = nullptr, const IDSelector *sel = nullptr)

Return the k nearest neighbors of each of the nx vectors x among the ny vector y, for the L2 distance

Parameters:

• x – query vectors, size nx * d

• y – database vectors, size ny * d

• res – result heap strcture, which also provides k. Sorted on output

• y_norm2 – (optional) norms for the y vectors (nullptr or size ny)

• sel – search in this subset of vectors

void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes, const float *y_norm2 = nullptr, const IDSelector *sel = nullptr)

Return the k nearest neighbors of each of the nx vectors x among the ny vector y, for the L2 distance

Parameters:

• x – query vectors, size nx * d

• y – database vectors, size ny * d

• distances – output distances, size nq * k

• indexes – output vector ids, size nq * k

• y_norm2 – (optional) norms for the y vectors (nullptr or size ny)

• sel – search in this subset of vectors

void knn_inner_products_by_idx(const float *x, const float *y, const int64_t *subset, size_t d, size_t nx, size_t ny, size_t nsubset, size_t k, float *vals, int64_t *ids, int64_t ld_ids = -1)

Find the max inner product neighbors for nx queries in a set of ny vectors indexed by ids. May be useful for re-ranking a pre-selected vector list

Parameters:

• x – query vectors, size nx * d

• y – database vectors, size (max(ids) + 1) * d

• ids – subset of database vectors to consider, size (nx, nsubset)

• res – result structure

• ld_ids – stride for the ids array. -1: use nsubset, 0: all queries process the same subset

void knn_L2sqr_by_idx(const float *x, const float *y, const int64_t *subset, size_t d, size_t nx, size_t ny, size_t nsubset, size_t k, float *vals, int64_t *ids, int64_t ld_subset = -1)

Find the nearest neighbors for nx queries in a set of ny vectors indexed by ids. May be useful for re-ranking a pre-selected vector list

Parameters:

• x – query vectors, size nx * d

• y – database vectors, size (max(ids) + 1) * d

• subset – subset of database vectors to consider, size (nx, nsubset)

• res – rIDesult structure

• ld_subset – stride for the subset array. -1: use nsubset, 0: all queries process the same subset

void range_search_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result, const IDSelector *sel = nullptr)

Return the k nearest neighbors of each of the nx vectors x among the ny vector y, w.r.t to max inner product

Parameters:

• x – query vectors, size nx * d

• y – database vectors, size ny * d

• radius – search radius around the x vectors

• result – result structure

void range_search_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result, const IDSelector *sel = nullptr)

same as range_search_L2sqr for the inner product similarity

void compute_PQ_dis_tables_dsub2(size_t d, size_t ksub, const float *centroids, size_t nx, const float *x, bool is_inner_product, float *dis_tables)

specialized function for PQ2

void fvec_madd(size_t n, const float *a, float bf, const float *b, float *c)

compute c := a + bf * b for a, b and c tables

Parameters:

• n – size of the tables

• a – size n

• b – size n

• c – result table, size n

int fvec_madd_and_argmin(size_t n, const float *a, float bf, const float *b, float *c)

same as fvec_madd, also return index of the min of the result table

Returns:

index of the min of table c

bool exhaustive_L2sqr_fused_cmax(const float *x, const float *y, size_t d, size_t nx, size_t ny, Top1BlockResultHandler<CMax<float, int64_t>> &res, const float *y_norms)

template<class Consumer, class ...Types>
Consumer::T dispatch_VectorDistance(size_t d, MetricType metric, float metric_arg, Consumer &consumer, Types... args)

void pairwise_extra_distances(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, MetricType mt, float metric_arg, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)

void knn_extra_metrics(const float *x, const float *y, size_t d, size_t nx, size_t ny, MetricType mt, float metric_arg, size_t k, float *distances, int64_t *indexes)

FlatCodesDistanceComputer *get_extra_distance_computer(size_t d, MetricType mt, float metric_arg, size_t nb, const float *xb)

get a DistanceComputer that refers to this type of distance and indexes a flat array of size nb

inline uint16_t encode_fp16(float x)

inline float decode_fp16(uint16_t x)

void bitvec_print(const uint8_t *b, size_t d)

void fvecs2bitvecs(const float *x, uint8_t *b, size_t d, size_t n)

void bitvecs2fvecs(const uint8_t *b, float *x, size_t d, size_t n)

void fvec2bitvec(const float *x, uint8_t *b, size_t d)

void bitvec_shuffle(size_t n, size_t da, size_t db, const int *order, const uint8_t *a, uint8_t *b)

Shuffle the bits from b(i, j) := a(i, order[j])

void hammings(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t nbytespercode, hamdis_t *dis)

Compute a set of Hamming distances between na and nb binary vectors

Parameters:

• a – size na * nbytespercode

• b – size nb * nbytespercode

• nbytespercode – should be multiple of 8

• dis – output distances, size na * nb

void hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered, ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK)

Return the k smallest Hamming distances for a set of binary query vectors, using a max heap.

Parameters:

• a – queries, size ha->nh * ncodes

• b – database, size nb * ncodes

• nb – number of database vectors

• ncodes – size of the binary codes (bytes)

• ordered – if != 0: order the results by decreasing distance (may be bottleneck for k/n > 0.01)

• approx_topk_mode – allows to use approximate top-k facilities to speedup heap

void hammings_knn(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered)

void hammings_knn_mc(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t k, size_t ncodes, int32_t *distances, int64_t *labels)

Return the k smallest Hamming distances for a set of binary query vectors, using counting max.

Parameters:

• a – queries, size na * ncodes

• b – database, size nb * ncodes

• na – number of query vectors

• nb – number of database vectors

• k – number of vectors/distances to return

• ncodes – size of the binary codes (bytes)

• distances – output distances from each query vector to its k nearest neighbors

• labels – output ids of the k nearest neighbors to each query vector

void hamming_range_search(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, int radius, size_t ncodes, RangeSearchResult *result)

same as hammings_knn except we are doing a range search with radius

void hamming_count_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, size_t *nptr)

size_t match_hamming_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, int64_t *idx, hamdis_t *dis)

void crosshamming_count_thres(const uint8_t *dbs, size_t n, hamdis_t ht, size_t ncodes, size_t *nptr)

inline hamdis_t hamming(const uint64_t *bs1, const uint64_t *bs2, size_t nwords)

void generalized_hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t code_size, int ordered = true)

generalized Hamming distances (= count number of code bytes that are the same)

void pack_bitstrings(size_t n, size_t M, int nbit, const int32_t *unpacked, uint8_t *packed, size_t code_size)

Pack a set of n codes of size M * nbit

Parameters:

• n – number of codes to pack

• M – number of elementary codes per code

• nbit – number of bits per elementary code

• unpacked – input unpacked codes, size (n, M)

• packed – output packed codes, size (n, code_size)

• code_size – should be >= ceil(M * nbit / 8)

void pack_bitstrings(size_t n, size_t M, const int32_t *nbits, const int32_t *unpacked, uint8_t *packed, size_t code_size)

Pack a set of n codes of variable sizes

Parameters:

nbit – number of bits per entry (size M)

void unpack_bitstrings(size_t n, size_t M, int nbit, const uint8_t *packed, size_t code_size, int32_t *unpacked)

Unpack a set of n codes of size M * nbit

Parameters:

• n – number of codes to pack

• M – number of elementary codes per code

• nbit – number of bits per elementary code

• unpacked – input unpacked codes, size (n, M)

• packed – output packed codes, size (n, code_size)

• code_size – should be >= ceil(M * nbit / 8)

void unpack_bitstrings(size_t n, size_t M, const int32_t *nbits, const uint8_t *packed, size_t code_size, int32_t *unpacked)

Unpack a set of n codes of variable sizes

Parameters:

nbit – number of bits per entry (size M)

template<size_t nbits, typename T>
inline T hamming(const uint8_t *bs1, const uint8_t *bs2)

template<size_t nbits>
inline hamdis_t hamming(const uint64_t *bs1, const uint64_t *bs2)

template<>
inline hamdis_t hamming<64>(const uint64_t *pa, const uint64_t *pb)

template<>
inline hamdis_t hamming<128>(const uint64_t *pa, const uint64_t *pb)

template<>
inline hamdis_t hamming<256>(const uint64_t *pa, const uint64_t *pb)

inline int generalized_hamming_64(uint64_t a)

inline int popcount32(uint32_t x)

inline int popcount64(uint64_t x)

SPECIALIZED_HC (4)

SPECIALIZED_HC (8)

SPECIALIZED_HC (16)

SPECIALIZED_HC (20)

SPECIALIZED_HC (32)

SPECIALIZED_HC (64)

template<class Consumer, class ...Types>
Consumer::T dispatch_HammingComputer(int code_size, Consumer &consumer, Types... args)

template<class C>
inline void heap_pop(size_t k, typename C::T *bh_val, typename C::TI *bh_ids)

Pops the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1]. on output the element at k-1 is undefined.

template<class C>
inline void heap_push(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id)

Pushes the element (val, ids) into the heap bh_val[0..k-2] and bh_ids[0..k-2]. on output the element at k-1 is defined.

template<class C>
inline void heap_replace_top(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id)

Replaces the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1], and for identical bh_val[] values also sorts by bh_ids[] values.

template<typename T>
inline void minheap_pop(size_t k, T *bh_val, int64_t *bh_ids)

template<typename T>
inline void minheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)

template<typename T>
inline void minheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)

template<typename T>
inline void maxheap_pop(size_t k, T *bh_val, int64_t *bh_ids)

template<typename T>
inline void maxheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)

template<typename T>
inline void maxheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)

template<class C>
inline void heap_pop(size_t k, std::pair<typename C::T, typename C::TI> *bh)

Pops the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1]. on output the element at k-1 is undefined.

template<class C>
inline void heap_push(size_t k, std::pair<typename C::T, typename C::TI> *bh, typename C::T val, typename C::TI id)

Pushes the element (val, ids) into the heap bh_val[0..k-2] and bh_ids[0..k-2]. on output the element at k-1 is defined.

template<class C>
inline void heap_replace_top(size_t k, std::pair<typename C::T, typename C::TI> *bh, typename C::T val, typename C::TI id)

Replaces the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1], and for identical bh_val[] values also sorts by bh_ids[] values.

template<class C>
inline void heap_heapify(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x = nullptr, const typename C::TI *ids = nullptr, size_t k0 = 0)

template<typename T>
inline void minheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)

template<typename T>
inline void maxheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)

template<class C>
inline void heap_addn(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x, const typename C::TI *ids, size_t n)

template<typename T>
inline void minheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)

template<typename T>
inline void maxheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)

template<typename C>
inline size_t heap_reorder(size_t k, typename C::T *bh_val, typename C::TI *bh_ids)

template<typename T>
inline size_t minheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)

template<typename T>
inline size_t maxheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)

template<class C>
inline void indirect_heap_pop(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids)

template<class C>
inline void indirect_heap_push(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids, typename C::TI id)

template<class idx_t, class C>
void merge_knn_results(size_t n, size_t k, typename C::TI nshard, const typename C::T *all_distances, const idx_t *all_labels, typename C::T *distances, idx_t *labels)

Merge result tables from several shards. The per-shard results are assumed to be sorted. Note that the C comparator is reversed w.r.t. the usual top-k element heap because we want the best (ie. lowest for L2) result to be on top, not the worst. Also, it needs to hold an index of a shard id (ie. usually int32 is more than enough).

Parameters:

• all_distances – size (nshard, n, k)

• all_labels – size (nshard, n, k)

• distances – output distances, size (n, k)

• labels – output labels, size (n, k)

template<typename T>
inline T cmin_nextafter(T x)

template<typename T>
inline T cmax_nextafter(T x)

template<>
inline float cmin_nextafter<float>(float x)

template<>
inline float cmax_nextafter<float>(float x)

template<>
inline uint16_t cmin_nextafter<uint16_t>(uint16_t x)

template<>
inline uint16_t cmax_nextafter<uint16_t>(uint16_t x)

template<class C>
C::T partition_fuzzy(typename C::T *vals, typename C::TI *ids, size_t n, size_t q_min, size_t q_max, size_t *q_out)

partitions the table into 0:q and q:n where all elements above q are >= all elements below q (for C = CMax, for CMin comparisons are reversed)

Returns the partition threshold. The elements q:n are destroyed on output.

template<class C>
inline C::T partition(typename C::T *vals, typename C::TI *ids, size_t n, size_t q)

simplified interface for when the parition is not fuzzy

void simd_histogram_8(const uint16_t *data, int n, uint16_t min, int shift, int *hist)

low level SIMD histogramming functions 8-bin histogram of (x - min) >> shift values outside the range are ignored. the data table should be aligned on 32 bytes

void simd_histogram_16(const uint16_t *data, int n, uint16_t min, int shift, int *hist)

same for 16-bin histogram

void float_rand(float *x, size_t n, int64_t seed)

void float_randn(float *x, size_t n, int64_t seed)

void int64_rand(int64_t *x, size_t n, int64_t seed)

void byte_rand(uint8_t *x, size_t n, int64_t seed)

void int64_rand_max(int64_t *x, size_t n, uint64_t max, int64_t seed)

void rand_perm(int *perm, size_t n, int64_t seed)

void rand_perm_splitmix64(int *perm, size_t n, int64_t seed)

void rand_smooth_vectors(size_t n, size_t d, float *x, int64_t seed)

inline simd16uint16 min(simd16uint16 a, simd16uint16 b)

inline simd16uint16 max(simd16uint16 a, simd16uint16 b)

inline simd16uint16 combine2x2(simd16uint16 a, simd16uint16 b)

inline uint32_t cmp_ge32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)

inline uint32_t cmp_le32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)

inline simd16uint16 hadd(const simd16uint16 &a, const simd16uint16 &b)

inline void cmplt_min_max_fast(const simd16uint16 candidateValues, const simd16uint16 candidateIndices, const simd16uint16 currentValues, const simd16uint16 currentIndices, simd16uint16 &minValues, simd16uint16 &minIndices, simd16uint16 &maxValues, simd16uint16 &maxIndices)

inline simd32uint8 uint16_to_uint8_saturate(simd16uint16 a, simd16uint16 b)

inline uint32_t get_MSBs(simd32uint8 a)

get most significant bit of each byte

inline simd32uint8 blendv(simd32uint8 a, simd32uint8 b, simd32uint8 mask)

use MSB of each byte of mask to select a byte between a and b

inline void cmplt_min_max_fast(const simd8uint32 candidateValues, const simd8uint32 candidateIndices, const simd8uint32 currentValues, const simd8uint32 currentIndices, simd8uint32 &minValues, simd8uint32 &minIndices, simd8uint32 &maxValues, simd8uint32 &maxIndices)

inline simd8float32 hadd(simd8float32 a, simd8float32 b)

inline simd8float32 unpacklo(simd8float32 a, simd8float32 b)

inline simd8float32 unpackhi(simd8float32 a, simd8float32 b)

inline simd8float32 fmadd(simd8float32 a, simd8float32 b, simd8float32 c)

inline void cmplt_and_blend_inplace(const simd8float32 candidateValues, const simd8uint32 candidateIndices, simd8float32 &lowestValues, simd8uint32 &lowestIndices)

inline void cmplt_min_max_fast(const simd8float32 candidateValues, const simd8uint32 candidateIndices, const simd8float32 currentValues, const simd8uint32 currentIndices, simd8float32 &minValues, simd8uint32 &minIndices, simd8float32 &maxValues, simd8uint32 &maxIndices)

inline simd16uint16 combine4x2(simd32uint16 a, simd32uint16 b)

inline simd16uint16 min(const simd16uint16 &av, const simd16uint16 &bv)

inline simd16uint16 max(const simd16uint16 &av, const simd16uint16 &bv)

inline simd16uint16 combine2x2(const simd16uint16 &a, const simd16uint16 &b)

inline uint32_t cmp_ge32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)

inline uint32_t cmp_le32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)

inline simd32uint8 uint16_to_uint8_saturate(const simd16uint16 &a, const simd16uint16 &b)

inline uint32_t get_MSBs(const simd32uint8 &a)

get most significant bit of each byte

inline simd32uint8 blendv(const simd32uint8 &a, const simd32uint8 &b, const simd32uint8 &mask)

use MSB of each byte of mask to select a byte between a and b

inline simd8float32 hadd(const simd8float32 &a, const simd8float32 &b)

inline simd8float32 unpacklo(const simd8float32 &a, const simd8float32 &b)

inline simd8float32 unpackhi(const simd8float32 &a, const simd8float32 &b)

inline simd8float32 fmadd(const simd8float32 &a, const simd8float32 &b, const simd8float32 &c)

void fvec_argsort(size_t n, const float *vals, size_t *perm)

Indirect sort of a floating-point array

Parameters:

• n – size of the array

• vals – array to sort, size n

• perm – output: permutation of [0..n-1], st. vals[perm[i + 1]] >= vals[perm[i]]

void fvec_argsort_parallel(size_t n, const float *vals, size_t *perm)

Same as fvec_argsort, parallelized

void bucket_sort(size_t nval, const uint64_t *vals, uint64_t nbucket, int64_t *lims, int64_t *perm, int nt = 0)

Bucket sort of a list of values

Parameters:

• vals – values to sort, size nval, max value nbucket - 1

• lims – output limits of buckets, size nbucket + 1

• perm – output buckets, the elements of bucket i are in perm[lims[i]:lims[i + 1]]

• nt – number of threads (0 = pure sequential code)

void matrix_bucket_sort_inplace(size_t nrow, size_t ncol, int32_t *vals, int32_t nbucket, int64_t *lims, int nt = 0)

in-place bucket sort (with attention to memory=>int32) on input the values are in a nrow * col matrix we want to store the row numbers in the output.

Parameters:

• vals – positive values to sort, size nrow * ncol, max value nbucket - 1

• lims – output limits of buckets, size nbucket + 1

• nt – number of threads (0 = pure sequential code)

void matrix_bucket_sort_inplace(size_t nrow, size_t ncol, int64_t *vals, int64_t nbucket, int64_t *lims, int nt = 0)

same with int64 elements

void hashtable_int64_to_int64_init(int log2_capacity, int64_t *tab)

Hashtable implementation for int64 -> int64 with external storage implemented for fast batch add and lookup.

tab is of size 2 * (1 << log2_capacity) n is the number of elements to add or search

adding several values in a same batch: an arbitrary one gets added in different batches: the newer batch overwrites. raises an exception if capacity is exhausted.

void hashtable_int64_to_int64_add(int log2_capacity, int64_t *tab, size_t n, const int64_t *keys, const int64_t *vals)

void hashtable_int64_to_int64_lookup(int log2_capacity, const int64_t *tab, size_t n, const int64_t *keys, int64_t *vals)

std::string get_compile_options()

get compile options

std::string get_version()

double getmillisecs()

ms elapsed since some arbitrary epoch

size_t get_mem_usage_kb()

get current RSS usage in kB

uint64_t get_cycles()

void reflection(const float *u, float *x, size_t n, size_t d, size_t nu)

void matrix_qr(int m, int n, float *a)

compute the Q of the QR decomposition for m > n

Parameters:

a – size n * m: input matrix and output Q

void ranklist_handle_ties(int k, int64_t *idx, const float *dis)

distances are supposed to be sorted. Sorts indices with same distance

size_t ranklist_intersection_size(size_t k1, const int64_t *v1, size_t k2, const int64_t *v2)

count the number of common elements between v1 and v2 algorithm = sorting + bissection to avoid double-counting duplicates

size_t merge_result_table_with(size_t n, size_t k, int64_t *I0, float *D0, const int64_t *I1, const float *D1, bool keep_min = true, int64_t translation = 0)

merge a result table into another one

Parameters:

• I0, D0 – first result table, size (n, k)

• I1, D1 – second result table, size (n, k)

• keep_min – if true, keep min values, otherwise keep max

• translation – add this value to all I1’s indexes

Returns:

nb of values that were taken from the second table

double imbalance_factor(int n, int k, const int64_t *assign)

a balanced assignment has a IF of 1

double imbalance_factor(int k, const int *hist)

same, takes a histogram as input

int ivec_hist(size_t n, const int *v, int vmax, int *hist)

compute histogram on v

void bincode_hist(size_t n, size_t nbits, const uint8_t *codes, int *hist)

Compute histogram of bits on a code array

Parameters:

• codes – size(n, nbits / 8)

• hist – size(nbits): nb of 1s in the array of codes

uint64_t ivec_checksum(size_t n, const int32_t *a)

compute a checksum on a table.

uint64_t bvec_checksum(size_t n, const uint8_t *a)

compute a checksum on a table.

void bvecs_checksum(size_t n, size_t d, const uint8_t *a, uint64_t *cs)

compute checksums for the rows of a matrix

Parameters:

• n – number of rows

• d – size per row

• a – matrix to handle, size n * d

• cs – output checksums, size n

const float *fvecs_maybe_subsample(size_t d, size_t *n, size_t nmax, const float *x, bool verbose = false, int64_t seed = 1234)

random subsamples a set of vectors if there are too many of them

Parameters:

• d – dimension of the vectors

• n – on input: nb of input vectors, output: nb of output vectors

• nmax – max nb of vectors to keep

• x – input array, size *n-by-d

• seed – random seed to use for sampling

Returns:

x or an array allocated with new [] with *n vectors

void binary_to_real(size_t d, const uint8_t *x_in, float *x_out)

Convert binary vector to +1/-1 valued float vector.

Parameters:

• d – dimension of the vector (multiple of 8)

• x_in – input binary vector (uint8_t table of size d / 8)

• x_out – output float vector (float table of size d)

void real_to_binary(size_t d, const float *x_in, uint8_t *x_out)

Convert float vector to binary vector. Components > 0 are converted to 1, others to 0.

Parameters:

• d – dimension of the vector (multiple of 8)

• x_in – input float vector (float table of size d)

• x_out – output binary vector (uint8_t table of size d / 8)

uint64_t hash_bytes(const uint8_t *bytes, int64_t n)

A reasonable hashing function

bool check_openmp()

Whether OpenMP annotations were respected.

Variables

FAISS_API HNSWStats hnsw_stats

FAISS_API int product_quantizer_compute_codes_bs

FAISS_API int distance_compute_min_k_reservoir

FAISS_API bool simd_result_handlers_accept_virtual

FAISS_API int index2layer_sa_encode_bs

FAISS_API int index_factory_verbose

set to > 0 to get more logs from index_factory

const int IO_FLAG_SKIP_STORAGE = 1

skip the storage for graph-based indexes

const int IO_FLAG_READ_ONLY = 2

const int IO_FLAG_ONDISK_SAME_DIR = 4

const int IO_FLAG_SKIP_IVF_DATA = 8

const int IO_FLAG_SKIP_PRECOMPUTE_TABLE = 16

const int IO_FLAG_PQ_SKIP_SDC_TABLE = 32

const int IO_FLAG_MMAP = IO_FLAG_SKIP_IVF_DATA | 0x646f0000

FAISS_API IndexBinaryHashStats indexBinaryHash_stats

FAISS_API FastScanStats FastScan_stats

FAISS_API bool check_compatible_for_merge_expensive_check

FAISS_API IndexIVFStats indexIVF_stats

FAISS_API IVFFastScanStats IVFFastScan_stats

FAISS_API size_t precomputed_table_max_bytes

FAISS_API int index_ivfpq_add_core_o_bs

FAISS_API IndexIVFPQStats indexIVFPQ_stats

FAISS_API IndexPQStats indexPQ_stats

FAISS_API int multi_index_quantizer_search_bs

FAISS_API int rowwise_minmax_sa_encode_bs

block size for performing sa_encode and sa_decode

FAISS_API int rowwise_minmax_sa_decode_bs

FAISS_API int distance_compute_blas_threshold

FAISS_API int distance_compute_blas_query_bs

FAISS_API int distance_compute_blas_database_bs

FAISS_API size_t hamming_batch_size

static constexpr uint8_t hamdis_tab_ham_bytes[256] = {0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8}

FAISS_API PartitionStats partition_stats

FAISS_API int bucket_sort_verbose

increase verbosity of the bucket_sort functions

struct AutoTuneCriterion

#include <AutoTune.h>

Evaluation criterion. Returns a performance measure in [0,1], higher is better.

Subclassed by faiss::IntersectionCriterion, faiss::OneRecallAtRCriterion

Public Functions

AutoTuneCriterion(idx_t nq, idx_t nnn)

void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)

Intitializes the gt_D and gt_I vectors. Must be called before evaluating

Parameters:

• gt_D_in – size nq * gt_nnn

• gt_I_in – size nq * gt_nnn

virtual double evaluate(const float *D, const idx_t *I) const = 0

Evaluate the criterion.

Parameters:

• D – size nq * nnn

• I – size nq * nnn

Returns:

the criterion, between 0 and 1. Larger is better.

inline virtual ~AutoTuneCriterion()

Public Members

idx_t nq

nb of queries this criterion is evaluated on

idx_t nnn

nb of NNs that the query should request

idx_t gt_nnn

nb of GT NNs required to evaluate criterion

std::vector<float> gt_D

Ground-truth distances (size nq * gt_nnn)

std::vector<idx_t> gt_I

Ground-truth indexes (size nq * gt_nnn)

struct OneRecallAtRCriterion : public faiss::AutoTuneCriterion

Public Functions

OneRecallAtRCriterion(idx_t nq, idx_t R)

virtual double evaluate(const float *D, const idx_t *I) const override

Evaluate the criterion.

Parameters:

• D – size nq * nnn

• I – size nq * nnn

Returns:

the criterion, between 0 and 1. Larger is better.

inline ~OneRecallAtRCriterion() override

void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)

Intitializes the gt_D and gt_I vectors. Must be called before evaluating

Parameters:

• gt_D_in – size nq * gt_nnn

• gt_I_in – size nq * gt_nnn

Public Members

idx_t R

idx_t nq

nb of queries this criterion is evaluated on

idx_t nnn

nb of NNs that the query should request

idx_t gt_nnn

nb of GT NNs required to evaluate criterion

std::vector<float> gt_D

Ground-truth distances (size nq * gt_nnn)

std::vector<idx_t> gt_I

Ground-truth indexes (size nq * gt_nnn)

struct IntersectionCriterion : public faiss::AutoTuneCriterion

Public Functions

IntersectionCriterion(idx_t nq, idx_t R)

virtual double evaluate(const float *D, const idx_t *I) const override

Evaluate the criterion.

Parameters:

• D – size nq * nnn

• I – size nq * nnn

Returns:

the criterion, between 0 and 1. Larger is better.

inline ~IntersectionCriterion() override

void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)

Intitializes the gt_D and gt_I vectors. Must be called before evaluating

Parameters:

• gt_D_in – size nq * gt_nnn

• gt_I_in – size nq * gt_nnn

Public Members

idx_t R

idx_t nq

nb of queries this criterion is evaluated on

idx_t nnn

nb of NNs that the query should request

idx_t gt_nnn

nb of GT NNs required to evaluate criterion

std::vector<float> gt_D

Ground-truth distances (size nq * gt_nnn)

std::vector<idx_t> gt_I

Ground-truth indexes (size nq * gt_nnn)

struct OperatingPoint

#include <AutoTune.h>

Maintains a list of experimental results. Each operating point is a (perf, t, key) triplet, where higher perf and lower t is better. The key field is an arbitrary identifier for the operating point.

Includes primitives to extract the Pareto-optimal operating points in the (perf, t) space.

Public Members

double perf

performance measure (output of a Criterion)

double t

corresponding execution time (ms)

std::string key

key that identifies this op pt

int64_t cno

integer identifier

struct OperatingPoints

Public Functions

OperatingPoints()

int merge_with(const OperatingPoints &other, const std::string &prefix = "")

add operating points from other to this, with a prefix to the keys

void clear()

bool add(double perf, double t, const std::string &key, size_t cno = 0)

add a performance measure. Return whether it is an optimal point

double t_for_perf(double perf) const

get time required to obtain a given performance measure

void display(bool only_optimal = true) const

easy-to-read output

void all_to_gnuplot(const char *fname) const

output to a format easy to digest by gnuplot

void optimal_to_gnuplot(const char *fname) const

Public Members

std::vector<OperatingPoint> all_pts

all operating points

std::vector<OperatingPoint> optimal_pts

optimal operating points, sorted by perf

struct ParameterRange

#include <AutoTune.h>

possible values of a parameter, sorted from least to most expensive/accurate

Public Members

std::string name

std::vector<double> values

struct ParameterSpace

#include <AutoTune.h>

Uses a-priori knowledge on the Faiss indexes to extract tunable parameters.

Subclassed by faiss::gpu::GpuParameterSpace

Public Functions

ParameterSpace()

size_t n_combinations() const

nb of combinations, = product of values sizes

bool combination_ge(size_t c1, size_t c2) const

returns whether combinations c1 >= c2 in the tuple sense

std::string combination_name(size_t cno) const

get string representation of the combination

void display() const

print a description on stdout

ParameterRange &add_range(const std::string &name)

add a new parameter (or return it if it exists)

virtual void initialize(const Index *index)

initialize with reasonable parameters for the index

void set_index_parameters(Index *index, size_t cno) const

set a combination of parameters on an index

void set_index_parameters(Index *index, const char *param_string) const

set a combination of parameters described by a string

virtual void set_index_parameter(Index *index, const std::string &name, double val) const

set one of the parameters, returns whether setting was successful

void update_bounds(size_t cno, const OperatingPoint &op, double *upper_bound_perf, double *lower_bound_t) const

find an upper bound on the performance and a lower bound on t for configuration cno given another operating point op

void explore(Index *index, size_t nq, const float *xq, const AutoTuneCriterion &crit, OperatingPoints *ops) const

explore operating points

Parameters:

• index – index to run on

• xq – query vectors (size nq * index.d)

• crit – selection criterion

• ops – resulting operating points

inline virtual ~ParameterSpace()

Public Members

std::vector<ParameterRange> parameter_ranges

all tunable parameters

int verbose

verbosity during exploration

int n_experiments

nb of experiments during optimization (0 = try all combinations)

size_t batchsize

maximum number of queries to submit at a time.

bool thread_over_batches

use multithreading over batches (useful to benchmark independent single-searches)

double min_test_duration

run tests several times until they reach at least this duration (to avoid jittering in MT mode)

struct Cloner

#include <clone_index.h>

Cloner class, useful to override classes with other cloning functions. The cloning function above just calls Cloner::clone_Index.

Subclassed by faiss::gpu::ToCPUCloner, faiss::gpu::ToGpuCloner, faiss::gpu::ToGpuClonerMultiple

Public Functions

virtual VectorTransform *clone_VectorTransform(const VectorTransform*)

virtual Index *clone_Index(const Index*)

virtual IndexIVF *clone_IndexIVF(const IndexIVF*)

inline virtual ~Cloner()

struct ClusteringParameters

#include <Clustering.h>

Class for the clustering parameters. Can be passed to the constructor of the Clustering object.

Subclassed by faiss::Clustering, faiss::ProgressiveDimClusteringParameters

Public Members

int niter = 25

number of clustering iterations

int nredo = 1

redo clustering this many times and keep the clusters with the best objective

bool verbose = false

bool spherical = false

whether to normalize centroids after each iteration (useful for inner product clustering)

bool int_centroids = false

round centroids coordinates to integer after each iteration?

bool update_index = false

re-train index after each iteration?

bool frozen_centroids = false

Use the subset of centroids provided as input and do not change them during iterations

int min_points_per_centroid = 39

If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.

int max_points_per_centroid = 256

to limit size of dataset, otherwise the training set is subsampled

int seed = 1234

seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.

size_t decode_block_size = 32768

when the training set is encoded, batch size of the codec decoder

bool check_input_data_for_NaNs = true

whether to check for NaNs in an input data

bool use_faster_subsampling = false

Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.

struct ClusteringIterationStats

Public Members

float obj

objective values (sum of distances reported by index)

double time

seconds for iteration

double time_search

seconds for just search

double imbalance_factor

imbalance factor of iteration

int nsplit

number of cluster splits

struct Clustering : public faiss::ClusteringParameters

#include <Clustering.h>

K-means clustering based on assignment - centroid update iterations

The clustering is based on an Index object that assigns training points to the centroids. Therefore, at each iteration the centroids are added to the index.

On output, the centoids table is set to the latest version of the centroids and they are also added to the index. If the centroids table it is not empty on input, it is also used for initialization.

Subclassed by faiss::Clustering1D

Public Functions

Clustering(int d, int k)

Clustering(int d, int k, const ClusteringParameters &cp)

virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)

run k-means training

Parameters:

• x – training vectors, size n * d

• index – index used for assignment

• x_weights – weight associated to each vector: NULL or size n

void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)

run with encoded vectors

win addition to train()’s parameters takes a codec as parameter to decode the input vectors.

Parameters:

codec – codec used to decode the vectors (nullptr = vectors are in fact floats)

void post_process_centroids()

Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding

inline virtual ~Clustering()

Public Members

size_t d

dimension of the vectors

size_t k

nb of centroids

std::vector<float> centroids

centroids (k * d) if centroids are set on input to train, they will be used as initialization

std::vector<ClusteringIterationStats> iteration_stats

stats at every iteration of clustering

int niter = 25

number of clustering iterations

int nredo = 1

redo clustering this many times and keep the clusters with the best objective

bool verbose = false

bool spherical = false

whether to normalize centroids after each iteration (useful for inner product clustering)

bool int_centroids = false

round centroids coordinates to integer after each iteration?

bool update_index = false

re-train index after each iteration?

bool frozen_centroids = false

Use the subset of centroids provided as input and do not change them during iterations

int min_points_per_centroid = 39

If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.

int max_points_per_centroid = 256

to limit size of dataset, otherwise the training set is subsampled

int seed = 1234

seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.

size_t decode_block_size = 32768

when the training set is encoded, batch size of the codec decoder

bool check_input_data_for_NaNs = true

whether to check for NaNs in an input data

bool use_faster_subsampling = false

Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.

struct Clustering1D : public faiss::Clustering

#include <Clustering.h>

Exact 1D clustering algorithm

Since it does not use an index, it does not overload the train() function

Public Functions

explicit Clustering1D(int k)

Clustering1D(int k, const ClusteringParameters &cp)

void train_exact(idx_t n, const float *x)

inline virtual ~Clustering1D()

virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)

run k-means training

Parameters:

• x – training vectors, size n * d

• index – index used for assignment

• x_weights – weight associated to each vector: NULL or size n

void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)

run with encoded vectors

win addition to train()’s parameters takes a codec as parameter to decode the input vectors.

Parameters:

codec – codec used to decode the vectors (nullptr = vectors are in fact floats)

void post_process_centroids()

Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding

Public Members

size_t d

dimension of the vectors

size_t k

nb of centroids

std::vector<float> centroids

centroids (k * d) if centroids are set on input to train, they will be used as initialization

std::vector<ClusteringIterationStats> iteration_stats

stats at every iteration of clustering

int niter = 25

number of clustering iterations

int nredo = 1

redo clustering this many times and keep the clusters with the best objective

bool verbose = false

bool spherical = false

whether to normalize centroids after each iteration (useful for inner product clustering)

bool int_centroids = false

round centroids coordinates to integer after each iteration?

bool update_index = false

re-train index after each iteration?

bool frozen_centroids = false

Use the subset of centroids provided as input and do not change them during iterations

int min_points_per_centroid = 39

If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.

int max_points_per_centroid = 256

to limit size of dataset, otherwise the training set is subsampled

int seed = 1234

seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.

size_t decode_block_size = 32768

when the training set is encoded, batch size of the codec decoder

bool check_input_data_for_NaNs = true

whether to check for NaNs in an input data

bool use_faster_subsampling = false

Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.

struct ProgressiveDimClusteringParameters : public faiss::ClusteringParameters

Subclassed by faiss::ProgressiveDimClustering

Public Functions

ProgressiveDimClusteringParameters()

Public Members

int progressive_dim_steps

number of incremental steps

bool apply_pca

apply PCA on input

int niter = 25

number of clustering iterations

int nredo = 1

redo clustering this many times and keep the clusters with the best objective

bool verbose = false

bool spherical = false

whether to normalize centroids after each iteration (useful for inner product clustering)

bool int_centroids = false

round centroids coordinates to integer after each iteration?

bool update_index = false

re-train index after each iteration?

bool frozen_centroids = false

Use the subset of centroids provided as input and do not change them during iterations

int min_points_per_centroid = 39

If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.

int max_points_per_centroid = 256

to limit size of dataset, otherwise the training set is subsampled

int seed = 1234

seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.

size_t decode_block_size = 32768

when the training set is encoded, batch size of the codec decoder

bool check_input_data_for_NaNs = true

whether to check for NaNs in an input data

bool use_faster_subsampling = false

Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.

struct ProgressiveDimIndexFactory

#include <Clustering.h>

generates an index suitable for clustering when called

Subclassed by faiss::gpu::GpuProgressiveDimIndexFactory

Public Functions

virtual Index *operator()(int dim)

ownership transferred to caller

inline virtual ~ProgressiveDimIndexFactory()

struct ProgressiveDimClustering : public faiss::ProgressiveDimClusteringParameters

#include <Clustering.h>

K-means clustering with progressive dimensions used

The clustering first happens in dim 1, then with exponentially increasing dimension until d (I steps). This is typically applied after a PCA transformation (optional). Reference:

“Improved Residual Vector Quantization for High-dimensional Approximate

Nearest Neighbor Search”

Shicong Liu, Hongtao Lu, Junru Shao, AAAI’15

https://arxiv.org/abs/1509.05195

Public Functions

ProgressiveDimClustering(int d, int k)

ProgressiveDimClustering(int d, int k, const ProgressiveDimClusteringParameters &cp)

void train(idx_t n, const float *x, ProgressiveDimIndexFactory &factory)

inline virtual ~ProgressiveDimClustering()

Public Members

size_t d

dimension of the vectors

size_t k

nb of centroids

std::vector<float> centroids

centroids (k * d)

std::vector<ClusteringIterationStats> iteration_stats

stats at every iteration of clustering

int progressive_dim_steps

number of incremental steps

bool apply_pca

apply PCA on input

int niter = 25

number of clustering iterations

int nredo = 1

redo clustering this many times and keep the clusters with the best objective

bool verbose = false

bool spherical = false

whether to normalize centroids after each iteration (useful for inner product clustering)

bool int_centroids = false

round centroids coordinates to integer after each iteration?

bool update_index = false

re-train index after each iteration?

bool frozen_centroids = false

Use the subset of centroids provided as input and do not change them during iterations

int min_points_per_centroid = 39

If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.

int max_points_per_centroid = 256

to limit size of dataset, otherwise the training set is subsampled

int seed = 1234

seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.

size_t decode_block_size = 32768

when the training set is encoded, batch size of the codec decoder

bool check_input_data_for_NaNs = true

whether to check for NaNs in an input data

bool use_faster_subsampling = false

Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.

struct AdditiveQuantizer : public faiss::Quantizer

#include <AdditiveQuantizer.h>

Abstract structure for additive quantizers

Different from the product quantizer in which the decoded vector is the concatenation of M sub-vectors, additive quantizers sum M sub-vectors to get the decoded vector.

Subclassed by faiss::LocalSearchQuantizer, faiss::ProductAdditiveQuantizer, faiss::ResidualQuantizer

Public Types

enum Search_type_t

Encodes how search is performed and how vectors are encoded.

Values:

enumerator ST_decompress

decompress database vector

enumerator ST_LUT_nonorm

use a LUT, don’t include norms (OK for IP or normalized vectors)

enumerator ST_norm_from_LUT

compute the norms from the look-up tables (cost is in O(M^2))

enumerator ST_norm_float

use a LUT, and store float32 norm with the vectors

enumerator ST_norm_qint8

use a LUT, and store 8bit-quantized norm

enumerator ST_norm_qint4

enumerator ST_norm_cqint8

use a LUT, and store non-uniform quantized norm

enumerator ST_norm_cqint4

enumerator ST_norm_lsq2x4

use a 2x4 bits lsq as norm quantizer (for fast scan)

enumerator ST_norm_rq2x4

use a 2x4 bits rq as norm quantizer (for fast scan)

Public Functions

void compute_codebook_tables()

uint64_t encode_norm(float norm) const

encode a norm into norm_bits bits

uint32_t encode_qcint(float x) const

encode norm by non-uniform scalar quantization

float decode_qcint(uint32_t c) const

decode norm by non-uniform scalar quantization

AdditiveQuantizer(size_t d, const std::vector<size_t> &nbits, Search_type_t search_type = ST_decompress)

AdditiveQuantizer()

compute derived values when d, M and nbits have been set

void set_derived_values()

Train the norm quantizer.

void train_norm(size_t n, const float *norms)

inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

Quantize a set of vectors

Parameters:

• x – input vectors, size n * d

• codes – output codes, size n * code_size

virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const = 0

Encode a set of vectors

Parameters:

• x – vectors to encode, size n * d

• codes – output codes, size n * code_size

• centroids – centroids to be added to x, size n * d

void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const

pack a series of code to bit-compact format

Parameters:

• codes – codes to be packed, size n * code_size

• packed_codes – output bit-compact codes

• ld_codes – leading dimension of codes

• norms – norms of the vectors (size n). Will be computed if needed but not provided

• centroids – centroids to be added to x, size n * d

virtual void decode(const uint8_t *codes, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – codes to decode, size n * code_size

• x – output vectors, size n * d

virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const

Decode a set of vectors in non-packed format

Parameters:

• codes – codes to decode, size n * ld_codes

• x – output vectors, size n * d

template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const

void decode_64bit(idx_t n, float *x) const

decoding function for a code in a 64-bit word

virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const

Compute inner-product look-up tables. Used in the centroid search functions.

Parameters:

• xq – query vector, size (n, d)

• LUT – look-up table, size (n, total_codebook_size)

• alpha – compute alpha * inner-product

• ld_lut – leading dimension of LUT

void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const

exact IP search

void compute_centroid_norms(float *norms) const

For L2 search we need the L2 norms of the centroids

Parameters:

norms – output norms table, size total_codebook_size

void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const

Exact L2 search, with precomputed norms

virtual ~AdditiveQuantizer()

virtual void train(size_t n, const float *x) = 0

Train the quantizer

Parameters:

x – training vectors, size n * d

Public Members

size_t M

number of codebooks

std::vector<size_t> nbits

bits for each step

std::vector<float> codebooks

codebooks

std::vector<uint64_t> codebook_offsets

codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d

size_t tot_bits = 0

total number of bits (indexes + norms)

size_t norm_bits = 0

bits allocated for the norms

size_t total_codebook_size = 0

size of the codebook in vectors

bool only_8bit = false

are all nbits = 8 (use faster decoder)

bool verbose = false

verbose during training?

bool is_trained = false

is trained or not

std::vector<float> norm_tabs

auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan

IndexFlat1D qnorm

store and search norms

std::vector<float> centroid_norms

norms of all codebook entries (size total_codebook_size)

std::vector<float> codebook_cross_products

dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)

size_t max_mem_distances = 5 * (size_t(1) << 30)

norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated

Search_type_t search_type

Also determines what’s in the codes.

float norm_min = NAN

min/max for quantization of norms

float norm_max = NAN

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct RangeSearchResult

#include <AuxIndexStructures.h>

The objective is to have a simple result structure while minimizing the number of mem copies in the result. The method do_allocation can be overloaded to allocate the result tables in the matrix type of a scripting language like Lua or Python.

Public Functions

explicit RangeSearchResult(size_t nq, bool alloc_lims = true)

lims must be allocated on input to range_search.

virtual void do_allocation()

called when lims contains the nb of elements result entries for each query

virtual ~RangeSearchResult()

Public Members

size_t nq

nb of queries

size_t *lims

size (nq + 1)

idx_t *labels

result for query i is labels[lims[i]:lims[i+1]]

float *distances

corresponding distances (not sorted)

size_t buffer_size

size of the result buffers used

struct BufferList

#include <AuxIndexStructures.h>

List of temporary buffers used to store results before they are copied to the RangeSearchResult object.

Subclassed by faiss::RangeSearchPartialResult

Public Functions

explicit BufferList(size_t buffer_size)

~BufferList()

void append_buffer()

create a new buffer

void add(idx_t id, float dis)

add one result, possibly appending a new buffer if needed

void copy_range(size_t ofs, size_t n, idx_t *dest_ids, float *dest_dis)

copy elemnts ofs:ofs+n-1 seen as linear data in the buffers to tables dest_ids, dest_dis

Public Members

size_t buffer_size

std::vector<Buffer> buffers

size_t wp

write pointer in the last buffer.

struct Buffer

Public Members

idx_t *ids

float *dis

struct RangeQueryResult

#include <AuxIndexStructures.h>

result structure for a single query

Public Functions

void add(float dis, idx_t id)

called by search function to report a new result

Public Members

idx_t qno

size_t nres

RangeSearchPartialResult *pres

struct RangeSearchPartialResult : public faiss::BufferList

#include <AuxIndexStructures.h>

the entries in the buffers are split per query

Public Functions

explicit RangeSearchPartialResult(RangeSearchResult *res_in)

eventually the result will be stored in res_in

RangeQueryResult &new_result(idx_t qno)

begin a new result

void finalize()

void set_lims()

called by range_search before do_allocation

void copy_result(bool incremental = false)

called by range_search after do_allocation

void append_buffer()

create a new buffer

void add(idx_t id, float dis)

add one result, possibly appending a new buffer if needed

void copy_range(size_t ofs, size_t n, idx_t *dest_ids, float *dest_dis)

copy elemnts ofs:ofs+n-1 seen as linear data in the buffers to tables dest_ids, dest_dis

Public Members

RangeSearchResult *res

std::vector<RangeQueryResult> queries

query ids + nb of results per query.

size_t buffer_size

std::vector<Buffer> buffers

size_t wp

write pointer in the last buffer.

Public Static Functions

static void merge(std::vector<RangeSearchPartialResult*> &partial_results, bool do_delete = true)

merge a set of PartialResult’s into one RangeSearchResult on output the partialresults are empty!

struct InterruptCallback

Subclassed by faiss::TimeoutCallback

Public Functions

virtual bool want_interrupt() = 0

inline virtual ~InterruptCallback()

Public Static Functions

static void clear_instance()

static void check()

check if:

• an interrupt callback is set

• the callback returns true if this is the case, then throw an exception. Should not be called from multiple threads.

static bool is_interrupted()

same as check() but return true if is interrupted instead of throwing. Can be called from multiple threads.

static size_t get_period_hint(size_t flops)

assuming each iteration takes a certain number of flops, what is a reasonable interval to check for interrupts?

Public Static Attributes

static std::mutex lock

static std::unique_ptr<InterruptCallback> instance

struct TimeoutCallback : public faiss::InterruptCallback

Public Functions

virtual bool want_interrupt() override

void set_timeout(double timeout_in_seconds)

Public Members

std::chrono::time_point<std::chrono::steady_clock> start

double timeout

Public Static Functions

static void reset(double timeout_in_seconds)

static void clear_instance()

static void check()

check if:

• an interrupt callback is set

• the callback returns true if this is the case, then throw an exception. Should not be called from multiple threads.

static bool is_interrupted()

same as check() but return true if is interrupted instead of throwing. Can be called from multiple threads.

static size_t get_period_hint(size_t flops)

assuming each iteration takes a certain number of flops, what is a reasonable interval to check for interrupts?

Public Static Attributes

static std::mutex lock

static std::unique_ptr<InterruptCallback> instance

struct VisitedTable

#include <AuxIndexStructures.h>

set implementation optimized for fast access.

Public Functions

inline explicit VisitedTable(int size)

inline void set(int no)

set flag #no to true

inline bool get(int no) const

get flag #no

inline void advance()

reset all flags to false

Public Members

std::vector<uint8_t> visited

uint8_t visno

struct CodePacker

#include <CodePacker.h>

Packing consists in combining a fixed number of codes of constant size (code_size) into a block of data where they may (or may not) be interleaved for efficient consumption by distance computation kernels. This exists for the “fast_scan” indexes on CPU and for some GPU kernels.

Subclassed by faiss::CodePackerFlat, faiss::CodePackerPQ4

Public Functions

virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const = 0

virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const = 0

virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const

virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const

inline virtual ~CodePacker()

Public Members

size_t code_size

size_t nvec

size_t block_size

struct CodePackerFlat : public faiss::CodePacker

#include <CodePacker.h>

Trivial code packer where codes are stored one by one

Public Functions

explicit CodePackerFlat(size_t code_size)

virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const final

virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const final

virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const final

virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const final

Public Members

size_t code_size

size_t nvec

size_t block_size

struct DistanceComputer

Subclassed by faiss::FlatCodesDistanceComputer, faiss::NegativeDistanceComputer

Public Functions

virtual void set_query(const float *x) = 0

called before computing distances. Pointer x should remain valid while operator () is called

virtual float operator()(idx_t i) = 0

compute distance of vector i to current query

inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3)

compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.

virtual float symmetric_dis(idx_t i, idx_t j) = 0

compute distance between two stored vectors

inline virtual ~DistanceComputer()

struct NegativeDistanceComputer : public faiss::DistanceComputer

Public Functions

inline explicit NegativeDistanceComputer(DistanceComputer *basedis)

inline virtual void set_query(const float *x) override

called before computing distances. Pointer x should remain valid while operator () is called

inline virtual float operator()(idx_t i) override

compute distance of vector i to current query

inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3) override

compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.

inline virtual float symmetric_dis(idx_t i, idx_t j) override

compute distance between two stored vectors

inline virtual ~NegativeDistanceComputer()

Public Members

DistanceComputer *basedis

owned by this

struct FlatCodesDistanceComputer : public faiss::DistanceComputer

Subclassed by faiss::ScalarQuantizer::SQDistanceComputer

Public Functions

inline FlatCodesDistanceComputer(const uint8_t *codes, size_t code_size)

inline FlatCodesDistanceComputer()

inline virtual float operator()(idx_t i) override

compute distance of vector i to current query

virtual float distance_to_code(const uint8_t *code) = 0

compute distance of current query to an encoded vector

inline virtual ~FlatCodesDistanceComputer()

virtual void set_query(const float *x) = 0

called before computing distances. Pointer x should remain valid while operator () is called

inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3)

compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.

virtual float symmetric_dis(idx_t i, idx_t j) = 0

compute distance between two stored vectors

Public Members

const uint8_t *codes

size_t code_size

class FaissException : public std::exception

#include <FaissException.h>

Base class for Faiss exceptions.

Public Functions

explicit FaissException(const std::string &msg)

FaissException(const std::string &msg, const char *funcName, const char *file, int line)

const char *what() const noexcept override

from std::exception

Public Members

std::string msg

struct TransformedVectors

#include <FaissException.h>

RAII object for a set of possibly transformed vectors (deallocated only if they are indeed transformed)

Public Functions

inline TransformedVectors(const float *x_orig, const float *x)

inline ~TransformedVectors()

Public Members

const float *x

bool own_x

template<class C>
struct ResultHandler

Subclassed by faiss::HeapBlockResultHandler< C, use_sel >::SingleResultHandler, faiss::RangeSearchBlockResultHandler< C, use_sel >::SingleResultHandler, faiss::ReservoirTopN< C >, faiss::Top1BlockResultHandler< C, use_sel >::SingleResultHandler

Public Functions

virtual bool add_result(typename C::T dis, typename C::TI idx) = 0

inline virtual ~ResultHandler()

Public Members

C::T threshold = C::neutral()

struct SearchParametersHNSW : public faiss::SearchParameters

Public Functions

inline ~SearchParametersHNSW()

Public Members

int efSearch = 16

bool check_relative_distance = true

bool bounded_queue = true

IDSelector *sel = nullptr

if non-null, only these IDs will be considered during search.

struct HNSW

Public Types

using storage_idx_t = int32_t

internal storage of vectors (32 bits: this is expensive)

using C = CMax<float, int64_t>

typedef std::pair<float, storage_idx_t> Node

Public Functions

void set_default_probas(int M, float levelMult)

initialize the assign_probas and cum_nneighbor_per_level to have 2*M links on level 0 and M links on levels > 0

void set_nb_neighbors(int level_no, int n)

set nb of neighbors for this level (before adding anything)

int nb_neighbors(int layer_no) const

nb of neighbors for this level

int cum_nb_neighbors(int layer_no) const

cumumlative nb up to (and excluding) this level

void neighbor_range(idx_t no, int layer_no, size_t *begin, size_t *end) const

range of entries in the neighbors table of vertex no at layer_no

explicit HNSW(int M = 32)

only mandatory parameter: nb of neighbors

int random_level()

pick a random level for a new point

void fill_with_random_links(size_t n)

add n random levels to table (for debugging…)

void add_links_starting_from(DistanceComputer &ptdis, storage_idx_t pt_id, storage_idx_t nearest, float d_nearest, int level, omp_lock_t *locks, VisitedTable &vt, bool keep_max_size_level0 = false)

void add_with_locks(DistanceComputer &ptdis, int pt_level, int pt_id, std::vector<omp_lock_t> &locks, VisitedTable &vt, bool keep_max_size_level0 = false)

add point pt_id on all levels <= pt_level and build the link structure for them.

HNSWStats search(DistanceComputer &qdis, ResultHandler<C> &res, VisitedTable &vt, const SearchParametersHNSW *params = nullptr) const

search interface for 1 point, single thread

void search_level_0(DistanceComputer &qdis, ResultHandler<C> &res, idx_t nprobe, const storage_idx_t *nearest_i, const float *nearest_d, int search_type, HNSWStats &search_stats, VisitedTable &vt, const SearchParametersHNSW *params = nullptr) const

search only in level 0 from a given vertex

void reset()

void clear_neighbor_tables(int level)

void print_neighbor_stats(int level) const

int prepare_level_tab(size_t n, bool preset_levels = false)

void permute_entries(const idx_t *map)

Public Members

std::vector<double> assign_probas

assignment probability to each layer (sum=1)

std::vector<int> cum_nneighbor_per_level

number of neighbors stored per layer (cumulative), should not be changed after first add

std::vector<int> levels

level of each vector (base level = 1), size = ntotal

std::vector<size_t> offsets

offsets[i] is the offset in the neighbors array where vector i is stored size ntotal + 1

std::vector<storage_idx_t> neighbors

neighbors[offsets[i]:offsets[i+1]] is the list of neighbors of vector i for all levels. this is where all storage goes.

storage_idx_t entry_point = -1

entry point in the search structure (one of the points with maximum level

faiss::RandomGenerator rng

int max_level = -1

maximum level

int efConstruction = 40

expansion factor at construction time

int efSearch = 16

expansion factor at search time

bool check_relative_distance = true

during search: do we check whether the next best distance is good enough?

bool search_bounded_queue = true

use bounded queue during exploration

Public Static Functions

static void shrink_neighbor_list(DistanceComputer &qdis, std::priority_queue<NodeDistFarther> &input, std::vector<NodeDistFarther> &output, int max_size, bool keep_max_size_level0 = false)

struct MinimaxHeap

#include <HNSW.h>

Heap structure that allows fast

Public Types

typedef faiss::CMax<float, storage_idx_t> HC

Public Functions

inline explicit MinimaxHeap(int n)

void push(storage_idx_t i, float v)

float max() const

int size() const

void clear()

int pop_min(float *vmin_out = nullptr)

int count_below(float thresh)

Public Members

int n

int k

int nvalid

std::vector<storage_idx_t> ids

std::vector<float> dis

struct NodeDistCloser

#include <HNSW.h>

to sort pairs of (id, distance) from nearest to fathest or the reverse

Public Functions

inline NodeDistCloser(float d, int id)

inline bool operator<(const NodeDistCloser &obj1) const

Public Members

float d

int id

struct NodeDistFarther

Public Functions

inline NodeDistFarther(float d, int id)

inline bool operator<(const NodeDistFarther &obj1) const

Public Members

float d

int id

struct HNSWStats

Public Functions

inline void reset()

number of hops aka number of edges traversed

inline void combine(const HNSWStats &other)

Public Members

size_t n1 = 0

size_t n2 = 0

number of vectors searched

size_t ndis = 0

number of queries for which the candidate list is exhausted

size_t nhops = 0

number of distances computed

struct IDSelector

#include <IDSelector.h>

Encapsulates a set of ids to handle.

Subclassed by PyCallbackIDSelector, faiss::IDSelectorAll, faiss::IDSelectorAnd, faiss::IDSelectorArray, faiss::IDSelectorBatch, faiss::IDSelectorBitmap, faiss::IDSelectorNot, faiss::IDSelectorOr, faiss::IDSelectorRange, faiss::IDSelectorTranslated, faiss::IDSelectorXOr

Public Functions

virtual bool is_member(idx_t id) const = 0

inline virtual ~IDSelector()

struct IDSelectorRange : public faiss::IDSelector

#include <IDSelector.h>

ids between [imin, imax)

Public Functions

IDSelectorRange(idx_t imin, idx_t imax, bool assume_sorted = false)

virtual bool is_member(idx_t id) const final

void find_sorted_ids_bounds(size_t list_size, const idx_t *ids, size_t *jmin, size_t *jmax) const

for sorted ids, find the range of list indices where the valid ids are stored

inline ~IDSelectorRange() override

Public Members

idx_t imin

idx_t imax

bool assume_sorted

Assume that the ids to handle are sorted. In some cases this can speed up processing

struct IDSelectorArray : public faiss::IDSelector

#include <IDSelector.h>

Simple array of elements

is_member calls are very inefficient, but some operations can use the ids directly.

Public Functions

IDSelectorArray(size_t n, const idx_t *ids)

Construct with an array of ids to process

Parameters:

• n – number of ids to store

• ids – elements to store. The pointer should remain valid during IDSelectorArray’s lifetime

virtual bool is_member(idx_t id) const final

inline ~IDSelectorArray() override

Public Members

size_t n

const idx_t *ids

struct IDSelectorBatch : public faiss::IDSelector

#include <IDSelector.h>

Ids from a set.

Repetitions of ids in the indices set passed to the constructor does not hurt performance.

The hash function used for the bloom filter and GCC’s implementation of unordered_set are just the least significant bits of the id. This works fine for random ids or ids in sequences but will produce many hash collisions if lsb’s are always the same

Public Functions

IDSelectorBatch(size_t n, const idx_t *indices)

Construct with an array of ids to process

Parameters:

• n – number of ids to store

• ids – elements to store. The pointer can be released after construction

virtual bool is_member(idx_t id) const final

inline ~IDSelectorBatch() override

Public Members

std::unordered_set<idx_t> set

std::vector<uint8_t> bloom

int nbits

idx_t mask

struct IDSelectorBitmap : public faiss::IDSelector

#include <IDSelector.h>

One bit per element. Constructed with a bitmap, size ceil(n / 8).

Public Functions

IDSelectorBitmap(size_t n, const uint8_t *bitmap)

Construct with a binary mask

Parameters:

• n – size of the bitmap array

• bitmap – id will be selected iff id / 8 < n and bit number (i%8) of bitmap[floor(i / 8)] is 1.

virtual bool is_member(idx_t id) const final

inline ~IDSelectorBitmap() override

Public Members

size_t n

const uint8_t *bitmap

struct IDSelectorNot : public faiss::IDSelector

#include <IDSelector.h>

reverts the membership test of another selector

Public Functions

inline IDSelectorNot(const IDSelector *sel)

inline virtual bool is_member(idx_t id) const final

inline virtual ~IDSelectorNot()

Public Members

const IDSelector *sel

struct IDSelectorAll : public faiss::IDSelector

#include <IDSelector.h>

selects all entries (useful for benchmarking)

Public Functions

inline virtual bool is_member(idx_t id) const final

inline virtual ~IDSelectorAll()

struct IDSelectorAnd : public faiss::IDSelector

#include <IDSelector.h>

does an AND operation on the the two given IDSelector’s is_membership results.

Public Functions

inline IDSelectorAnd(const IDSelector *lhs, const IDSelector *rhs)

inline virtual bool is_member(idx_t id) const final

inline virtual ~IDSelectorAnd()

Public Members

const IDSelector *lhs

const IDSelector *rhs

struct IDSelectorOr : public faiss::IDSelector

#include <IDSelector.h>

does an OR operation on the the two given IDSelector’s is_membership results.

Public Functions

inline IDSelectorOr(const IDSelector *lhs, const IDSelector *rhs)

inline virtual bool is_member(idx_t id) const final

inline virtual ~IDSelectorOr()

Public Members

const IDSelector *lhs

const IDSelector *rhs

struct IDSelectorXOr : public faiss::IDSelector

#include <IDSelector.h>

does an XOR operation on the the two given IDSelector’s is_membership results.

Public Functions

inline IDSelectorXOr(const IDSelector *lhs, const IDSelector *rhs)

inline virtual bool is_member(idx_t id) const final

inline virtual ~IDSelectorXOr()

Public Members

const IDSelector *lhs

const IDSelector *rhs

struct IOReader

Subclassed by PyCallbackIOReader, faiss::BufferedIOReader, faiss::FileIOReader, faiss::VectorIOReader

Public Functions

virtual size_t operator()(void *ptr, size_t size, size_t nitems) = 0

virtual int filedescriptor()

inline virtual ~IOReader()

Public Members

std::string name

struct IOWriter

Subclassed by PyCallbackIOWriter, faiss::BufferedIOWriter, faiss::FileIOWriter, faiss::VectorIOWriter

Public Functions

virtual size_t operator()(const void *ptr, size_t size, size_t nitems) = 0

virtual int filedescriptor()

inline virtual ~IOWriter() noexcept(false)

Public Members

std::string name

struct VectorIOReader : public faiss::IOReader

Public Functions

virtual size_t operator()(void *ptr, size_t size, size_t nitems) override

virtual int filedescriptor()

Public Members

std::vector<uint8_t> data

size_t rp = 0

std::string name

struct VectorIOWriter : public faiss::IOWriter

Public Functions

virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override

virtual int filedescriptor()

Public Members

std::vector<uint8_t> data

std::string name

struct FileIOReader : public faiss::IOReader

Public Functions

FileIOReader(FILE *rf)

FileIOReader(const char *fname)

~FileIOReader() override

virtual size_t operator()(void *ptr, size_t size, size_t nitems) override

virtual int filedescriptor() override

Public Members

FILE *f = nullptr

bool need_close = false

std::string name

struct FileIOWriter : public faiss::IOWriter

Public Functions

FileIOWriter(FILE *wf)

FileIOWriter(const char *fname)

~FileIOWriter() override

virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override

virtual int filedescriptor() override

Public Members

FILE *f = nullptr

bool need_close = false

std::string name

struct BufferedIOReader : public faiss::IOReader

#include <io.h>

wraps an ioreader to make buffered reads to avoid too small reads

Public Functions

explicit BufferedIOReader(IOReader *reader, size_t bsz = 1024 * 1024)

Parameters:

bsz – buffer size (bytes). Reads will be done by batched of this size

virtual size_t operator()(void *ptr, size_t size, size_t nitems) override

virtual int filedescriptor()

Public Members

IOReader *reader

size_t bsz

size_t ofs

offset in input stream

size_t ofs2

number of bytes returned to caller

size_t b0

size_t b1

range of available bytes in the buffer

std::vector<char> buffer

std::string name

struct BufferedIOWriter : public faiss::IOWriter

Public Functions

explicit BufferedIOWriter(IOWriter *writer, size_t bsz = 1024 * 1024)

virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override

~BufferedIOWriter() override

virtual int filedescriptor()

Public Members

IOWriter *writer

size_t bsz

size_t ofs

size_t ofs2

number of bytes received from caller

size_t b0

amount of data in buffer

std::vector<char> buffer

std::string name

struct ZnSphereSearch

#include <lattice_Zn.h>

returns the nearest vertex in the sphere to a query. Returns only the coordinates, not an id.

Algorithm: all points are derived from a one atom vector up to a permutation and sign changes. The search function finds the most appropriate atom and transformation.

Subclassed by faiss::ZnSphereCodec

Public Functions

ZnSphereSearch(int dim, int r2)

float search(const float *x, float *c) const

find nearest centroid. x does not need to be normalized

float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const

full call. Requires externally-allocated temp space

void search_multi(int n, const float *x, float *c_out, float *dp_out)

Public Members

int dimS

int r2

int natom

std::vector<float> voc

size dim * ntatom

struct EnumeratedVectors

Subclassed by faiss::ZnSphereCodec, faiss::ZnSphereCodecRec

Public Functions

inline explicit EnumeratedVectors(int dim)

virtual uint64_t encode(const float *x) const = 0

encode a vector from a collection

virtual void decode(uint64_t code, float *c) const = 0

decode it

void encode_multi(size_t nc, const float *c, uint64_t *codes) const

void decode_multi(size_t nc, const uint64_t *codes, float *c) const

void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)

inline virtual ~EnumeratedVectors()

Public Members

uint64_t nv

size of the collection

int dim

struct Repeat

Public Members

float val

int n

struct Repeats

#include <lattice_Zn.h>

Repeats: used to encode a vector that has n occurrences of val. Encodes the signs and permutation of the vector. Useful for atoms.

Subclassed by faiss::ZnSphereCodec::CodeSegment

Public Functions

Repeats(int dim = 0, const float *c = nullptr)

uint64_t count() const

uint64_t encode(const float *c) const

void decode(uint64_t code, float *c) const

Public Members

int dim

std::vector<Repeat> repeats

struct ZnSphereCodec : public faiss::ZnSphereSearch, public faiss::EnumeratedVectors

#include <lattice_Zn.h>

codec that can return ids for the encoded vectors

uses the ZnSphereSearch to encode the vector by encoding the permutation and signs. Depends on ZnSphereSearch because it uses the atom numbers

Subclassed by faiss::ZnSphereCodecAlt

Public Functions

ZnSphereCodec(int dim, int r2)

uint64_t search_and_encode(const float *x) const

virtual void decode(uint64_t code, float *c) const override

decode it

virtual uint64_t encode(const float *x) const override

takes vectors that do not need to be centroids

float search(const float *x, float *c) const

find nearest centroid. x does not need to be normalized

float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const

full call. Requires externally-allocated temp space

void search_multi(int n, const float *x, float *c_out, float *dp_out)

void encode_multi(size_t nc, const float *c, uint64_t *codes) const

void decode_multi(size_t nc, const uint64_t *codes, float *c) const

void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)

Public Members

std::vector<CodeSegment> code_segments

uint64_t nv

size_t code_size

int dimS

int r2

int natom

std::vector<float> voc

size dim * ntatom

int dim

struct CodeSegment : public faiss::Repeats

Public Functions

inline explicit CodeSegment(const Repeats &r)

uint64_t count() const

uint64_t encode(const float *c) const

void decode(uint64_t code, float *c) const

Public Members

uint64_t c0

int signbits

int dim

std::vector<Repeat> repeats

struct ZnSphereCodecRec : public faiss::EnumeratedVectors

#include <lattice_Zn.h>

recursive sphere codec

Uses a recursive decomposition on the dimensions to encode centroids found by the ZnSphereSearch. The codes are not compatible with the ones of ZnSpehreCodec

Public Functions

ZnSphereCodecRec(int dim, int r2)

uint64_t encode_centroid(const float *c) const

virtual void decode(uint64_t code, float *c) const override

decode it

virtual uint64_t encode(const float *x) const override

vectors need to be centroids (does not work on arbitrary vectors)

uint64_t get_nv(int ld, int r2a) const

uint64_t get_nv_cum(int ld, int r2t, int r2a) const

void set_nv_cum(int ld, int r2t, int r2a, uint64_t v)

void encode_multi(size_t nc, const float *c, uint64_t *codes) const

void decode_multi(size_t nc, const uint64_t *codes, float *c) const

void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)

Public Members

int r2

int log2_dim

int code_size

std::vector<uint64_t> all_nv

std::vector<uint64_t> all_nv_cum

int decode_cache_ld

std::vector<std::vector<float>> decode_cache

uint64_t nv

size of the collection

int dim

struct ZnSphereCodecAlt : public faiss::ZnSphereCodec

#include <lattice_Zn.h>

Codec that uses the recursive codec if dim is a power of 2 and the regular one otherwise

Public Functions

ZnSphereCodecAlt(int dim, int r2)

virtual uint64_t encode(const float *x) const override

takes vectors that do not need to be centroids

virtual void decode(uint64_t code, float *c) const override

decode it

uint64_t search_and_encode(const float *x) const

float search(const float *x, float *c) const

find nearest centroid. x does not need to be normalized

float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const

full call. Requires externally-allocated temp space

void search_multi(int n, const float *x, float *c_out, float *dp_out)

void encode_multi(size_t nc, const float *c, uint64_t *codes) const

void decode_multi(size_t nc, const uint64_t *codes, float *c) const

void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)

Public Members

bool use_rec

ZnSphereCodecRec znc_rec

std::vector<CodeSegment> code_segments

uint64_t nv

size_t code_size

int dimS

int r2

int natom

std::vector<float> voc

size dim * ntatom

int dim

struct LocalSearchQuantizer : public faiss::AdditiveQuantizer

#include <LocalSearchQuantizer.h>

Implementation of LSQ/LSQ++ described in the following two papers:

Revisiting additive quantization Julieta Martinez, et al. ECCV 2016

LSQ++: Lower running time and higher recall in multi-codebook quantization Julieta Martinez, et al. ECCV 2018

This implementation is mostly translated from the Julia implementations by Julieta Martinez: (https://github.com/una-dinosauria/local-search-quantization, https://github.com/una-dinosauria/Rayuela.jl)

The trained codes are stored in codebooks which is called centroids in PQ and RQ.

Public Types

enum Search_type_t

Encodes how search is performed and how vectors are encoded.

Values:

enumerator ST_decompress

decompress database vector

enumerator ST_LUT_nonorm

use a LUT, don’t include norms (OK for IP or normalized vectors)

enumerator ST_norm_from_LUT

compute the norms from the look-up tables (cost is in O(M^2))

enumerator ST_norm_float

use a LUT, and store float32 norm with the vectors

enumerator ST_norm_qint8

use a LUT, and store 8bit-quantized norm

enumerator ST_norm_qint4

enumerator ST_norm_cqint8

use a LUT, and store non-uniform quantized norm

enumerator ST_norm_cqint4

enumerator ST_norm_lsq2x4

use a 2x4 bits lsq as norm quantizer (for fast scan)

enumerator ST_norm_rq2x4

use a 2x4 bits rq as norm quantizer (for fast scan)

Public Functions

LocalSearchQuantizer(size_t d, size_t M, size_t nbits, Search_type_t search_type = ST_decompress)

LocalSearchQuantizer()

~LocalSearchQuantizer() override

virtual void train(size_t n, const float *x) override

Train the quantizer

Parameters:

x – training vectors, size n * d

virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override

Encode a set of vectors

Parameters:

• x – vectors to encode, size n * d

• codes – output codes, size n * code_size

• n – number of vectors

• centroids – centroids to be added to x, size n * d

void update_codebooks(const float *x, const int32_t *codes, size_t n)

Update codebooks given encodings

Parameters:

• x – training vectors, size n * d

• codes – encoded training vectors, size n * M

• n – number of vectors

void icm_encode(int32_t *codes, const float *x, size_t n, size_t ils_iters, std::mt19937 &gen) const

Encode vectors given codebooks using iterative conditional mode (icm).

Parameters:

• codes – output codes, size n * M

• x – vectors to encode, size n * d

• n – number of vectors

• ils_iters – number of iterations of iterative local search

void icm_encode_impl(int32_t *codes, const float *x, const float *unaries, std::mt19937 &gen, size_t n, size_t ils_iters, bool verbose) const

void icm_encode_step(int32_t *codes, const float *unaries, const float *binaries, size_t n, size_t n_iters) const

void perturb_codes(int32_t *codes, size_t n, std::mt19937 &gen) const

Add some perturbation to codes

Parameters:

• codes – codes to be perturbed, size n * M

• n – number of vectors

void perturb_codebooks(float T, const std::vector<float> &stddev, std::mt19937 &gen)

Add some perturbation to codebooks

Parameters:

• T – temperature of simulated annealing

• stddev – standard derivations of each dimension in training data

void compute_binary_terms(float *binaries) const

Compute binary terms

Parameters:

binaries – binary terms, size M * M * K * K

void compute_unary_terms(const float *x, float *unaries, size_t n) const

Compute unary terms

Parameters:

• n – number of vectors

• x – vectors to encode, size n * d

• unaries – unary terms, size n * M * K

float evaluate(const int32_t *codes, const float *x, size_t n, float *objs = nullptr) const

Helper function to compute reconstruction error

Parameters:

• codes – encoded codes, size n * M

• x – vectors to encode, size n * d

• n – number of vectors

• objs – if it is not null, store reconstruction error of each vector into it, size n

void compute_codebook_tables()

uint64_t encode_norm(float norm) const

encode a norm into norm_bits bits

uint32_t encode_qcint(float x) const

encode norm by non-uniform scalar quantization

float decode_qcint(uint32_t c) const

decode norm by non-uniform scalar quantization

void set_derived_values()

Train the norm quantizer.

void train_norm(size_t n, const float *norms)

inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

Quantize a set of vectors

Parameters:

• x – input vectors, size n * d

• codes – output codes, size n * code_size

void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const

pack a series of code to bit-compact format

Parameters:

• codes – codes to be packed, size n * code_size

• packed_codes – output bit-compact codes

• ld_codes – leading dimension of codes

• norms – norms of the vectors (size n). Will be computed if needed but not provided

• centroids – centroids to be added to x, size n * d

virtual void decode(const uint8_t *codes, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – codes to decode, size n * code_size

• x – output vectors, size n * d

virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const

Decode a set of vectors in non-packed format

Parameters:

• codes – codes to decode, size n * ld_codes

• x – output vectors, size n * d

template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const

void decode_64bit(idx_t n, float *x) const

decoding function for a code in a 64-bit word

virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const

Compute inner-product look-up tables. Used in the centroid search functions.

Parameters:

• xq – query vector, size (n, d)

• LUT – look-up table, size (n, total_codebook_size)

• alpha – compute alpha * inner-product

• ld_lut – leading dimension of LUT

void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const

exact IP search

void compute_centroid_norms(float *norms) const

For L2 search we need the L2 norms of the centroids

Parameters:

norms – output norms table, size total_codebook_size

void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const

Exact L2 search, with precomputed norms

Public Members

size_t K

number of codes per codebook

size_t train_iters = 25

number of iterations in training

size_t encode_ils_iters = 16

iterations of local search in encoding

size_t train_ils_iters = 8

iterations of local search in training

size_t icm_iters = 4

number of iterations in icm

float p = 0.5f

temperature factor

float lambd = 1e-2f

regularization factor

size_t chunk_size = 10000

nb of vectors to encode at a time

int random_seed = 0x12345

seed for random generator

size_t nperts = 4

number of perturbation in each code

if non-NULL, use this encoder to encode (owned by the object)

lsq::IcmEncoderFactory *icm_encoder_factory = nullptr

bool update_codebooks_with_double = true

size_t M

number of codebooks

std::vector<size_t> nbits

bits for each step

std::vector<float> codebooks

codebooks

std::vector<uint64_t> codebook_offsets

codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d

size_t tot_bits = 0

total number of bits (indexes + norms)

size_t norm_bits = 0

bits allocated for the norms

size_t total_codebook_size = 0

size of the codebook in vectors

bool only_8bit = false

are all nbits = 8 (use faster decoder)

bool verbose = false

verbose during training?

bool is_trained = false

is trained or not

std::vector<float> norm_tabs

auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan

IndexFlat1D qnorm

store and search norms

std::vector<float> centroid_norms

norms of all codebook entries (size total_codebook_size)

std::vector<float> codebook_cross_products

dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)

size_t max_mem_distances = 5 * (size_t(1) << 30)

norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated

Search_type_t search_type

Also determines what’s in the codes.

float norm_min = NAN

min/max for quantization of norms

float norm_max = NAN

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct DummyScaler

#include <LookupTableScaler.h>

no-op handler

Public Functions

inline simd32uint8 lookup(const simd32uint8&, const simd32uint8&) const

inline simd16uint16 scale_lo(const simd32uint8&) const

inline simd16uint16 scale_hi(const simd32uint8&) const

template<class dist_t>
inline dist_t scale_one(const dist_t&) const

Public Static Attributes

static constexpr int nscale = 0

struct NormTableScaler

#include <LookupTableScaler.h>

consumes 2x4 bits to encode a norm as a scalar additive quantizer the norm is scaled because its range if larger than other components

Public Functions

inline explicit NormTableScaler(int scale)

inline simd32uint8 lookup(const simd32uint8 &lut, const simd32uint8 &c) const

inline simd16uint16 scale_lo(const simd32uint8 &res) const

inline simd16uint16 scale_hi(const simd32uint8 &res) const

template<class dist_t>
inline dist_t scale_one(const dist_t &x) const

Public Members

int scale_int

simd16uint16 scale_simd

Public Static Attributes

static constexpr int nscale = 2

struct NNDescent

Public Types

using storage_idx_t = int

using KNNGraph = std::vector<nndescent::Nhood>

Public Functions

explicit NNDescent(const int d, const int K)

~NNDescent()

void build(DistanceComputer &qdis, const int n, bool verbose)

void search(DistanceComputer &qdis, const int topk, idx_t *indices, float *dists, VisitedTable &vt) const

void reset()

void init_graph(DistanceComputer &qdis)

Initialize the KNN graph randomly.

void nndescent(DistanceComputer &qdis, bool verbose)

Perform NNDescent algorithm.

void join(DistanceComputer &qdis)

Perform local join on each node.

void update()

Sample new neighbors for each node to peform local join later.

void generate_eval_set(DistanceComputer &qdis, std::vector<int> &c, std::vector<std::vector<int>> &v, int N)

Sample a small number of points to evaluate the quality of KNNG built.

float eval_recall(std::vector<int> &ctrl_points, std::vector<std::vector<int>> &acc_eval_set)

Evaluate the quality of KNNG built.

Public Members

bool has_built = false

int S = 10

int R = 100

int iter = 10

int search_L = 0

int random_seed = 2021

int K

int d

int L

int ntotal = 0

KNNGraph graph

std::vector<int> final_graph

struct NSG

Public Types

using storage_idx_t = int32_t

internal storage of vectors (32 bits: this is expensive)

using Node = nsg::Node

using Neighbor = nsg::Neighbor

Public Functions

explicit NSG(int R = 32)

void build(Index *storage, idx_t n, const nsg::Graph<idx_t> &knn_graph, bool verbose)

void reset()

void search(DistanceComputer &dis, int k, idx_t *I, float *D, VisitedTable &vt) const

void init_graph(Index *storage, const nsg::Graph<idx_t> &knn_graph)

template<bool collect_fullset, class index_t>
void search_on_graph(const nsg::Graph<index_t> &graph, DistanceComputer &dis, VisitedTable &vt, int ep, int pool_size, std::vector<Neighbor> &retset, std::vector<Node> &fullset) const

void add_reverse_links(int q, std::vector<std::mutex> &locks, DistanceComputer &dis, nsg::Graph<Node> &graph)

void sync_prune(int q, std::vector<Node> &pool, DistanceComputer &dis, VisitedTable &vt, const nsg::Graph<idx_t> &knn_graph, nsg::Graph<Node> &graph)

void link(Index *storage, const nsg::Graph<idx_t> &knn_graph, nsg::Graph<Node> &graph, bool verbose)

int tree_grow(Index *storage, std::vector<int> &degrees)

int dfs(VisitedTable &vt, int root, int cnt) const

int attach_unlinked(Index *storage, VisitedTable &vt, VisitedTable &vt2, std::vector<int> &degrees)

void check_graph() const

Public Members

int ntotal = 0

nb of nodes

int R

nb of neighbors per node

int L

length of the search path at construction time

int C

candidate pool size at construction time

int search_L = 16

length of the search path

int enterpoint

enterpoint

std::shared_ptr<nsg::Graph<int32_t>> final_graph

NSG graph structure.

bool is_built = false

NSG is built or not.

RandomGenerator rng

random generator

struct SimulatedAnnealingParameters

#include <PolysemousTraining.h>

parameters used for the simulated annealing method

Subclassed by faiss::PolysemousTraining, faiss::SimulatedAnnealingOptimizer

Public Functions

inline SimulatedAnnealingParameters()

Public Members

double init_temperature = 0.7

double temperature_decay = 0.9997893011688015

int n_iter = 500000

int n_redo = 2

int seed = 123

int verbose = 0

bool only_bit_flips = false

bool init_random = false

struct PermutationObjective

#include <PolysemousTraining.h>

abstract class for the loss function

Subclassed by faiss::ReproduceDistancesObjective

Public Functions

virtual double compute_cost(const int *perm) const = 0

virtual double cost_update(const int *perm, int iw, int jw) const

inline virtual ~PermutationObjective()

Public Members

int n

struct ReproduceDistancesObjective : public faiss::PermutationObjective

Public Functions

double dis_weight(double x) const

double get_source_dis(int i, int j) const

virtual double compute_cost(const int *perm) const override

virtual double cost_update(const int *perm, int iw, int jw) const override

ReproduceDistancesObjective(int n, const double *source_dis_in, const double *target_dis_in, double dis_weight_factor)

void set_affine_target_dis(const double *source_dis_in)

inline ~ReproduceDistancesObjective() override

Public Members

double dis_weight_factor

std::vector<double> source_dis

“real” corrected distances (size n^2)

const double *target_dis

wanted distances (size n^2)

std::vector<double> weights

weights for each distance (size n^2)

int n

Public Static Functions

static inline double sqr(double x)

static void compute_mean_stdev(const double *tab, size_t n2, double *mean_out, double *stddev_out)

struct SimulatedAnnealingOptimizer : public faiss::SimulatedAnnealingParameters

#include <PolysemousTraining.h>

Simulated annealing optimization algorithm for permutations.

Public Functions

SimulatedAnnealingOptimizer(PermutationObjective *obj, const SimulatedAnnealingParameters &p)

logs values of the cost function

double optimize(int *perm)

double run_optimization(int *best_perm)

virtual ~SimulatedAnnealingOptimizer()

Public Members

PermutationObjective *obj

int n

size of the permutation

FILE *logfile

RandomGenerator *rnd

double init_cost

remember initial cost of optimization

double init_temperature = 0.7

double temperature_decay = 0.9997893011688015

int n_iter = 500000

int n_redo = 2

int seed = 123

int verbose = 0

bool only_bit_flips = false

bool init_random = false

struct PolysemousTraining : public faiss::SimulatedAnnealingParameters

#include <PolysemousTraining.h>

optimizes the order of indices in a ProductQuantizer

Public Types

enum Optimization_type_t

Values:

enumerator OT_None

enumerator OT_ReproduceDistances_affine

default

enumerator OT_Ranking_weighted_diff

same as _2, but use rank of y+ - rank of y-

Public Functions

PolysemousTraining()

void optimize_pq_for_hamming(ProductQuantizer &pq, size_t n, const float *x) const

reorder the centroids so that the Hamming distance becomes a good approximation of the SDC distance (called by train)

void optimize_ranking(ProductQuantizer &pq, size_t n, const float *x) const

called by optimize_pq_for_hamming

void optimize_reproduce_distances(ProductQuantizer &pq) const

called by optimize_pq_for_hamming

size_t memory_usage_per_thread(const ProductQuantizer &pq) const

make sure we don’t blow up the memory

Public Members

Optimization_type_t optimization_type

int ntrain_permutation

use 1/4 of the training points for the optimization, with max. ntrain_permutation. If ntrain_permutation == 0: train on centroids

double dis_weight_factor

decay of exp that weights distance loss

size_t max_memory

refuse to train if it would require more than that amount of RAM

std::string log_pattern

double init_temperature = 0.7

double temperature_decay = 0.9997893011688015

int n_iter = 500000

int n_redo = 2

int seed = 123

int verbose = 0

bool only_bit_flips = false

bool init_random = false

struct CodePackerPQ4 : public faiss::CodePacker

#include <pq4_fast_scan.h>

CodePacker API for the PQ4 fast-scan

Public Functions

CodePackerPQ4(size_t nsq, size_t bbs)

virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const final

virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const final

virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const

virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const

Public Members

size_t nsq

size_t code_size

size_t nvec

size_t block_size

struct ProductAdditiveQuantizer : public faiss::AdditiveQuantizer

#include <ProductAdditiveQuantizer.h>

Product Additive Quantizers

The product additive quantizer is a variant of AQ and PQ. It first splits the vector space into multiple orthogonal sub-spaces just like PQ does. And then it quantizes each sub-space by an independent additive quantizer.

Subclassed by faiss::ProductLocalSearchQuantizer, faiss::ProductResidualQuantizer

Public Types

enum Search_type_t

Encodes how search is performed and how vectors are encoded.

Values:

enumerator ST_decompress

decompress database vector

enumerator ST_LUT_nonorm

use a LUT, don’t include norms (OK for IP or normalized vectors)

enumerator ST_norm_from_LUT

compute the norms from the look-up tables (cost is in O(M^2))

enumerator ST_norm_float

use a LUT, and store float32 norm with the vectors

enumerator ST_norm_qint8

use a LUT, and store 8bit-quantized norm

enumerator ST_norm_qint4

enumerator ST_norm_cqint8

use a LUT, and store non-uniform quantized norm

enumerator ST_norm_cqint4

enumerator ST_norm_lsq2x4

use a 2x4 bits lsq as norm quantizer (for fast scan)

enumerator ST_norm_rq2x4

use a 2x4 bits rq as norm quantizer (for fast scan)

Public Functions

ProductAdditiveQuantizer(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type = ST_decompress)

Construct a product additive quantizer.

The additive quantizers passed in will be cloned into the ProductAdditiveQuantizer object.

Parameters:

• d – dimensionality of the input vectors

• aqs – sub-additive quantizers

• search_type – AQ search type

ProductAdditiveQuantizer()

virtual ~ProductAdditiveQuantizer()

void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)

AdditiveQuantizer *subquantizer(size_t m) const

Train the product additive quantizer.

virtual void train(size_t n, const float *x) override

Train the quantizer

Parameters:

x – training vectors, size n * d

virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override

Encode a set of vectors

Parameters:

• x – vectors to encode, size n * d

• codes – output codes, size n * code_size

• centroids – centroids to be added to x, size n * d

void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const

virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override

Decode a set of vectors in non-packed format

Parameters:

• codes – codes to decode, size n * ld_codes

• x – output vectors, size n * d

virtual void decode(const uint8_t *codes, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – codes to decode, size n * code_size

• x – output vectors, size n * d

virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override

Compute inner-product look-up tables. Used in the search functions.

Parameters:

• xq – query vector, size (n, d)

• LUT – look-up table, size (n, total_codebook_size)

• alpha – compute alpha * inner-product

• ld_lut – leading dimension of LUT

void compute_codebook_tables()

uint64_t encode_norm(float norm) const

encode a norm into norm_bits bits

uint32_t encode_qcint(float x) const

encode norm by non-uniform scalar quantization

float decode_qcint(uint32_t c) const

decode norm by non-uniform scalar quantization

void set_derived_values()

Train the norm quantizer.

void train_norm(size_t n, const float *norms)

inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

Quantize a set of vectors

Parameters:

• x – input vectors, size n * d

• codes – output codes, size n * code_size

void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const

pack a series of code to bit-compact format

Parameters:

• codes – codes to be packed, size n * code_size

• packed_codes – output bit-compact codes

• ld_codes – leading dimension of codes

• norms – norms of the vectors (size n). Will be computed if needed but not provided

• centroids – centroids to be added to x, size n * d

template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const

void decode_64bit(idx_t n, float *x) const

decoding function for a code in a 64-bit word

void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const

exact IP search

void compute_centroid_norms(float *norms) const

For L2 search we need the L2 norms of the centroids

Parameters:

norms – output norms table, size total_codebook_size

void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const

Exact L2 search, with precomputed norms

Public Members

size_t nsplits

number of sub-vectors we split a vector into

std::vector<AdditiveQuantizer*> quantizers

size_t M

number of codebooks

std::vector<size_t> nbits

bits for each step

std::vector<float> codebooks

codebooks

std::vector<uint64_t> codebook_offsets

codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d

size_t tot_bits = 0

total number of bits (indexes + norms)

size_t norm_bits = 0

bits allocated for the norms

size_t total_codebook_size = 0

size of the codebook in vectors

bool only_8bit = false

are all nbits = 8 (use faster decoder)

bool verbose = false

verbose during training?

bool is_trained = false

is trained or not

std::vector<float> norm_tabs

auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan

IndexFlat1D qnorm

store and search norms

std::vector<float> centroid_norms

norms of all codebook entries (size total_codebook_size)

std::vector<float> codebook_cross_products

dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)

size_t max_mem_distances = 5 * (size_t(1) << 30)

norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated

Search_type_t search_type

Also determines what’s in the codes.

float norm_min = NAN

min/max for quantization of norms

float norm_max = NAN

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct ProductLocalSearchQuantizer : public faiss::ProductAdditiveQuantizer

#include <ProductAdditiveQuantizer.h>

Product Local Search Quantizer

Public Types

enum Search_type_t

Encodes how search is performed and how vectors are encoded.

Values:

enumerator ST_decompress

decompress database vector

enumerator ST_LUT_nonorm

use a LUT, don’t include norms (OK for IP or normalized vectors)

enumerator ST_norm_from_LUT

compute the norms from the look-up tables (cost is in O(M^2))

enumerator ST_norm_float

use a LUT, and store float32 norm with the vectors

enumerator ST_norm_qint8

use a LUT, and store 8bit-quantized norm

enumerator ST_norm_qint4

enumerator ST_norm_cqint8

use a LUT, and store non-uniform quantized norm

enumerator ST_norm_cqint4

enumerator ST_norm_lsq2x4

use a 2x4 bits lsq as norm quantizer (for fast scan)

enumerator ST_norm_rq2x4

use a 2x4 bits rq as norm quantizer (for fast scan)

Public Functions

ProductLocalSearchQuantizer(size_t d, size_t nsplits, size_t Msub, size_t nbits, Search_type_t search_type = ST_decompress)

Construct a product LSQ object.

Parameters:

• d – dimensionality of the input vectors

• nsplits – number of sub-vectors we split a vector into

• Msub – number of codebooks of each LSQ

• nbits – bits for each step

• search_type – AQ search type

ProductLocalSearchQuantizer()

void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)

AdditiveQuantizer *subquantizer(size_t m) const

Train the product additive quantizer.

virtual void train(size_t n, const float *x) override

Train the quantizer

Parameters:

x – training vectors, size n * d

virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override

Encode a set of vectors

Parameters:

• x – vectors to encode, size n * d

• codes – output codes, size n * code_size

• centroids – centroids to be added to x, size n * d

void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const

virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override

Decode a set of vectors in non-packed format

Parameters:

• codes – codes to decode, size n * ld_codes

• x – output vectors, size n * d

virtual void decode(const uint8_t *codes, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – codes to decode, size n * code_size

• x – output vectors, size n * d

virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override

Compute inner-product look-up tables. Used in the search functions.

Parameters:

• xq – query vector, size (n, d)

• LUT – look-up table, size (n, total_codebook_size)

• alpha – compute alpha * inner-product

• ld_lut – leading dimension of LUT

void compute_codebook_tables()

uint64_t encode_norm(float norm) const

encode a norm into norm_bits bits

uint32_t encode_qcint(float x) const

encode norm by non-uniform scalar quantization

float decode_qcint(uint32_t c) const

decode norm by non-uniform scalar quantization

void set_derived_values()

Train the norm quantizer.

void train_norm(size_t n, const float *norms)

inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

Quantize a set of vectors

Parameters:

• x – input vectors, size n * d

• codes – output codes, size n * code_size

void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const

pack a series of code to bit-compact format

Parameters:

• codes – codes to be packed, size n * code_size

• packed_codes – output bit-compact codes

• ld_codes – leading dimension of codes

• norms – norms of the vectors (size n). Will be computed if needed but not provided

• centroids – centroids to be added to x, size n * d

template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const

void decode_64bit(idx_t n, float *x) const

decoding function for a code in a 64-bit word

void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const

exact IP search

void compute_centroid_norms(float *norms) const

For L2 search we need the L2 norms of the centroids

Parameters:

norms – output norms table, size total_codebook_size

void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const

Exact L2 search, with precomputed norms

Public Members

size_t nsplits

number of sub-vectors we split a vector into

std::vector<AdditiveQuantizer*> quantizers

size_t M

number of codebooks

std::vector<size_t> nbits

bits for each step

std::vector<float> codebooks

codebooks

std::vector<uint64_t> codebook_offsets

codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d

size_t tot_bits = 0

total number of bits (indexes + norms)

size_t norm_bits = 0

bits allocated for the norms

size_t total_codebook_size = 0

size of the codebook in vectors

bool only_8bit = false

are all nbits = 8 (use faster decoder)

bool verbose = false

verbose during training?

bool is_trained = false

is trained or not

std::vector<float> norm_tabs

auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan

IndexFlat1D qnorm

store and search norms

std::vector<float> centroid_norms

norms of all codebook entries (size total_codebook_size)

std::vector<float> codebook_cross_products

dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)

size_t max_mem_distances = 5 * (size_t(1) << 30)

norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated

Search_type_t search_type

Also determines what’s in the codes.

float norm_min = NAN

min/max for quantization of norms

float norm_max = NAN

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct ProductResidualQuantizer : public faiss::ProductAdditiveQuantizer

#include <ProductAdditiveQuantizer.h>

Product Residual Quantizer

Public Types

enum Search_type_t

Encodes how search is performed and how vectors are encoded.

Values:

enumerator ST_decompress

decompress database vector

enumerator ST_LUT_nonorm

use a LUT, don’t include norms (OK for IP or normalized vectors)

enumerator ST_norm_from_LUT

compute the norms from the look-up tables (cost is in O(M^2))

enumerator ST_norm_float

use a LUT, and store float32 norm with the vectors

enumerator ST_norm_qint8

use a LUT, and store 8bit-quantized norm

enumerator ST_norm_qint4

enumerator ST_norm_cqint8

use a LUT, and store non-uniform quantized norm

enumerator ST_norm_cqint4

enumerator ST_norm_lsq2x4

use a 2x4 bits lsq as norm quantizer (for fast scan)

enumerator ST_norm_rq2x4

use a 2x4 bits rq as norm quantizer (for fast scan)

Public Functions

ProductResidualQuantizer(size_t d, size_t nsplits, size_t Msub, size_t nbits, Search_type_t search_type = ST_decompress)

Construct a product RQ object.

Parameters:

• d – dimensionality of the input vectors

• nsplits – number of sub-vectors we split a vector into

• Msub – number of codebooks of each RQ

• nbits – bits for each step

• search_type – AQ search type

ProductResidualQuantizer()

void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)

AdditiveQuantizer *subquantizer(size_t m) const

Train the product additive quantizer.

virtual void train(size_t n, const float *x) override

Train the quantizer

Parameters:

x – training vectors, size n * d

virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override

Encode a set of vectors

Parameters:

• x – vectors to encode, size n * d

• codes – output codes, size n * code_size

• centroids – centroids to be added to x, size n * d

void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const

virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override

Decode a set of vectors in non-packed format

Parameters:

• codes – codes to decode, size n * ld_codes

• x – output vectors, size n * d

virtual void decode(const uint8_t *codes, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – codes to decode, size n * code_size

• x – output vectors, size n * d

virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override

Compute inner-product look-up tables. Used in the search functions.

Parameters:

• xq – query vector, size (n, d)

• LUT – look-up table, size (n, total_codebook_size)

• alpha – compute alpha * inner-product

• ld_lut – leading dimension of LUT

void compute_codebook_tables()

uint64_t encode_norm(float norm) const

encode a norm into norm_bits bits

uint32_t encode_qcint(float x) const

encode norm by non-uniform scalar quantization

float decode_qcint(uint32_t c) const

decode norm by non-uniform scalar quantization

void set_derived_values()

Train the norm quantizer.

void train_norm(size_t n, const float *norms)

inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

Quantize a set of vectors

Parameters:

• x – input vectors, size n * d

• codes – output codes, size n * code_size

void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const

pack a series of code to bit-compact format

Parameters:

• codes – codes to be packed, size n * code_size

• packed_codes – output bit-compact codes

• ld_codes – leading dimension of codes

• norms – norms of the vectors (size n). Will be computed if needed but not provided

• centroids – centroids to be added to x, size n * d

template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const

void decode_64bit(idx_t n, float *x) const

decoding function for a code in a 64-bit word

void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const

exact IP search

void compute_centroid_norms(float *norms) const

For L2 search we need the L2 norms of the centroids

Parameters:

norms – output norms table, size total_codebook_size

void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const

Exact L2 search, with precomputed norms

Public Members

size_t nsplits

number of sub-vectors we split a vector into

std::vector<AdditiveQuantizer*> quantizers

size_t M

number of codebooks

std::vector<size_t> nbits

bits for each step

std::vector<float> codebooks

codebooks

std::vector<uint64_t> codebook_offsets

codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d

size_t tot_bits = 0

total number of bits (indexes + norms)

size_t norm_bits = 0

bits allocated for the norms

size_t total_codebook_size = 0

size of the codebook in vectors

bool only_8bit = false

are all nbits = 8 (use faster decoder)

bool verbose = false

verbose during training?

bool is_trained = false

is trained or not

std::vector<float> norm_tabs

auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan

IndexFlat1D qnorm

store and search norms

std::vector<float> centroid_norms

norms of all codebook entries (size total_codebook_size)

std::vector<float> codebook_cross_products

dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)

size_t max_mem_distances = 5 * (size_t(1) << 30)

norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated

Search_type_t search_type

Also determines what’s in the codes.

float norm_min = NAN

min/max for quantization of norms

float norm_max = NAN

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct ProductQuantizer : public faiss::Quantizer

#include <ProductQuantizer.h>

Product Quantizer. PQ is trained using k-means, minimizing the L2 distance to centroids. PQ supports L2 and Inner Product search, however the quantization error is biased towards L2 distance.

Public Types

enum train_type_t

initialization

Values:

enumerator Train_default

enumerator Train_hot_start

the centroids are already initialized

enumerator Train_shared

share dictionary across PQ segments

enumerator Train_hypercube

initialize centroids with nbits-D hypercube

enumerator Train_hypercube_pca

initialize centroids with nbits-D hypercube

Public Functions

inline float *get_centroids(size_t m, size_t i)

return the centroids associated with subvector m

inline const float *get_centroids(size_t m, size_t i) const

virtual void train(size_t n, const float *x) override

Train the quantizer

Parameters:

x – training vectors, size n * d

ProductQuantizer(size_t d, size_t M, size_t nbits)

ProductQuantizer()

void set_derived_values()

compute derived values when d, M and nbits have been set

void set_params(const float *centroids, int m)

Define the centroids for subquantizer m.

void compute_code(const float *x, uint8_t *code) const

Quantize one vector with the product quantizer.

virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

same as compute_code for several vectors

void compute_codes_with_assign_index(const float *x, uint8_t *codes, size_t n)

speed up code assignment using assign_index (non-const because the index is changed)

void decode(const uint8_t *code, float *x) const

decode a vector from a given code (or n vectors if third argument)

virtual void decode(const uint8_t *code, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – input codes, size n * code_size

• x – output vectors, size n * d

void compute_code_from_distance_table(const float *tab, uint8_t *code) const

If we happen to have the distance tables precomputed, this is more efficient to compute the codes.

void compute_distance_table(const float *x, float *dis_table) const

Compute distance table for one vector.

The distance table for x = [x_0 x_1 .. x_(M-1)] is a M * ksub matrix that contains

dis_table (m, j) = || x_m - c_(m, j)||^2 for m = 0..M-1 and j = 0 .. ksub - 1

where c_(m, j) is the centroid no j of sub-quantizer m.

Parameters:

• x – input vector size d

• dis_table – output table, size M * ksub

void compute_inner_prod_table(const float *x, float *dis_table) const

void compute_distance_tables(size_t nx, const float *x, float *dis_tables) const

compute distance table for several vectors

Parameters:

• nx – nb of input vectors

• x – input vector size nx * d

• dis_table – output table, size nx * M * ksub

void compute_inner_prod_tables(size_t nx, const float *x, float *dis_tables) const

void search(const float *x, size_t nx, const uint8_t *codes, const size_t ncodes, float_maxheap_array_t *res, bool init_finalize_heap = true) const

perform a search (L2 distance)

Parameters:

• x – query vectors, size nx * d

• nx – nb of queries

• codes – database codes, size ncodes * code_size

• ncodes – nb of nb vectors

• res – heap array to store results (nh == nx)

• init_finalize_heap – initialize heap (input) and sort (output)?

void search_ip(const float *x, size_t nx, const uint8_t *codes, const size_t ncodes, float_minheap_array_t *res, bool init_finalize_heap = true) const

same as search, but with inner product similarity

void compute_sdc_table()

void search_sdc(const uint8_t *qcodes, size_t nq, const uint8_t *bcodes, const size_t ncodes, float_maxheap_array_t *res, bool init_finalize_heap = true) const

void sync_transposed_centroids()

Sync transposed centroids with regular centroids. This call is needed if centroids were edited directly.

void clear_transposed_centroids()

Clear transposed centroids table so ones are no longer used.

Public Members

size_t M

number of subquantizers

size_t nbits

number of bits per quantization index

size_t dsub

dimensionality of each subvector

size_t ksub

number of centroids for each subquantizer

bool verbose

verbose during training?

train_type_t train_type

ClusteringParameters cp

parameters used during clustering

Index *assign_index

if non-NULL, use this index for assignment (should be of size d / M)

std::vector<float> centroids

Centroid table, size M * ksub * dsub. Layout: (M, ksub, dsub)

std::vector<float> transposed_centroids

Transposed centroid table, size M * ksub * dsub. Layout: (dsub, M, ksub)

std::vector<float> centroids_sq_lengths

Squared lengths of centroids, size M * ksub Layout: (M, ksub)

std::vector<float> sdc_table

Symmetric Distance Table.

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct PQEncoderGeneric

Public Functions

inline PQEncoderGeneric(uint8_t *code, int nbits, uint8_t offset = 0)

inline void encode(uint64_t x)

inline ~PQEncoderGeneric()

Public Members

uint8_t *code

code for this vector

uint8_t offset

const int nbits

number of bits per subquantizer index

uint8_t reg

struct PQEncoder8

Public Functions

inline PQEncoder8(uint8_t *code, int nbits)

inline void encode(uint64_t x)

Public Members

uint8_t *code

struct PQEncoder16

Public Functions

inline PQEncoder16(uint8_t *code, int nbits)

inline void encode(uint64_t x)

Public Members

uint16_t *code

struct PQDecoderGeneric

Public Functions

inline PQDecoderGeneric(const uint8_t *code, int nbits)

inline uint64_t decode()

Public Members

const uint8_t *code

uint8_t offset

const int nbits

const uint64_t mask

uint8_t reg

struct PQDecoder8

Public Functions

inline PQDecoder8(const uint8_t *code, int nbits)

inline uint64_t decode()

Public Members

const uint8_t *code

Public Static Attributes

static const int nbits = 8

struct PQDecoder16

Public Functions

inline PQDecoder16(const uint8_t *code, int nbits)

inline uint64_t decode()

Public Members

const uint16_t *code

Public Static Attributes

static const int nbits = 16

struct Quantizer

#include <Quantizer.h>

General interface for quantizer objects

Subclassed by faiss::AdditiveQuantizer, faiss::ProductQuantizer, faiss::ScalarQuantizer

Public Functions

inline explicit Quantizer(size_t d = 0, size_t code_size = 0)

virtual void train(size_t n, const float *x) = 0

Train the quantizer

Parameters:

x – training vectors, size n * d

virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const = 0

Quantize a set of vectors

Parameters:

• x – input vectors, size n * d

• codes – output codes, size n * code_size

virtual void decode(const uint8_t *code, float *x, size_t n) const = 0

Decode a set of vectors

Parameters:

• codes – input codes, size n * code_size

• x – output vectors, size n * d

inline virtual ~Quantizer()

Public Members

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct ResidualQuantizer : public faiss::AdditiveQuantizer

#include <ResidualQuantizer.h>

Residual quantizer with variable number of bits per sub-quantizer

The residual centroids are stored in a big cumulative centroid table. The codes are represented either as a non-compact table of size (n, M) or as the compact output (n, code_size).

Public Types

using train_type_t = int

initialization

enum Search_type_t

Encodes how search is performed and how vectors are encoded.

Values:

enumerator ST_decompress

decompress database vector

enumerator ST_LUT_nonorm

use a LUT, don’t include norms (OK for IP or normalized vectors)

enumerator ST_norm_from_LUT

compute the norms from the look-up tables (cost is in O(M^2))

enumerator ST_norm_float

use a LUT, and store float32 norm with the vectors

enumerator ST_norm_qint8

use a LUT, and store 8bit-quantized norm

enumerator ST_norm_qint4

enumerator ST_norm_cqint8

use a LUT, and store non-uniform quantized norm

enumerator ST_norm_cqint4

enumerator ST_norm_lsq2x4

use a 2x4 bits lsq as norm quantizer (for fast scan)

enumerator ST_norm_rq2x4

use a 2x4 bits rq as norm quantizer (for fast scan)

Public Functions

ResidualQuantizer(size_t d, const std::vector<size_t> &nbits, Search_type_t search_type = ST_decompress)

ResidualQuantizer(size_t d, size_t M, size_t nbits, Search_type_t search_type = ST_decompress)

ResidualQuantizer()

virtual void train(size_t n, const float *x) override

Train the residual quantizer.

void initialize_from(const ResidualQuantizer &other, int skip_M = 0)

Copy the M codebook levels from other, starting from skip_M.

float retrain_AQ_codebook(size_t n, const float *x)

Encode the vectors and compute codebook that minimizes the quantization error on these codes

Parameters:

• x – training vectors, size n * d

• n – nb of training vectors, n >= total_codebook_size

Returns:

returns quantization error for the new codebook with old codes

virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override

Encode a set of vectors

Parameters:

• x – vectors to encode, size n * d

• codes – output codes, size n * code_size

• centroids – centroids to be added to x, size n * d

void refine_beam(size_t n, size_t beam_size, const float *residuals, int new_beam_size, int32_t *new_codes, float *new_residuals = nullptr, float *new_distances = nullptr) const

lower-level encode function

Parameters:

• n – number of vectors to handle

• residuals – vectors to encode, size (n, beam_size, d)

• beam_size – input beam size

• new_beam_size – output beam size (should be <= K * beam_size)

• new_codes – output codes, size (n, new_beam_size, m + 1)

• new_residuals – output residuals, size (n, new_beam_size, d)

• new_distances – output distances, size (n, new_beam_size)

void refine_beam_LUT(size_t n, const float *query_norms, const float *query_cp, int new_beam_size, int32_t *new_codes, float *new_distances = nullptr) const

size_t memory_per_point(int beam_size = -1) const

Beam search can consume a lot of memory. This function estimates the amount of mem used by refine_beam to adjust the batch size

Parameters:

beam_size – if != -1, override the beam size

void compute_codebook_tables()

uint64_t encode_norm(float norm) const

encode a norm into norm_bits bits

uint32_t encode_qcint(float x) const

encode norm by non-uniform scalar quantization

float decode_qcint(uint32_t c) const

decode norm by non-uniform scalar quantization

void set_derived_values()

Train the norm quantizer.

void train_norm(size_t n, const float *norms)

inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

Quantize a set of vectors

Parameters:

• x – input vectors, size n * d

• codes – output codes, size n * code_size

void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const

pack a series of code to bit-compact format

Parameters:

• codes – codes to be packed, size n * code_size

• packed_codes – output bit-compact codes

• ld_codes – leading dimension of codes

• norms – norms of the vectors (size n). Will be computed if needed but not provided

• centroids – centroids to be added to x, size n * d

virtual void decode(const uint8_t *codes, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – codes to decode, size n * code_size

• x – output vectors, size n * d

virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const

Decode a set of vectors in non-packed format

Parameters:

• codes – codes to decode, size n * ld_codes

• x – output vectors, size n * d

template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const

void decode_64bit(idx_t n, float *x) const

decoding function for a code in a 64-bit word

virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const

Compute inner-product look-up tables. Used in the centroid search functions.

Parameters:

• xq – query vector, size (n, d)

• LUT – look-up table, size (n, total_codebook_size)

• alpha – compute alpha * inner-product

• ld_lut – leading dimension of LUT

void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const

exact IP search

void compute_centroid_norms(float *norms) const

For L2 search we need the L2 norms of the centroids

Parameters:

norms – output norms table, size total_codebook_size

void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const

Exact L2 search, with precomputed norms

Public Members

train_type_t train_type = Train_progressive_dim

Binary or of the Train_* flags below.

int niter_codebook_refine = 5

number of iterations for codebook refinement.

int max_beam_size = 5

beam size used for training and for encoding

int use_beam_LUT = 0

use LUT for beam search

ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK

Currently used mode of approximate min-k computations. Default value is EXACT_TOPK.

ProgressiveDimClusteringParameters cp

clustering parameters

ProgressiveDimIndexFactory *assign_index_factory = nullptr

if non-NULL, use this index for assignment

size_t M

number of codebooks

std::vector<size_t> nbits

bits for each step

std::vector<float> codebooks

codebooks

std::vector<uint64_t> codebook_offsets

codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d

size_t tot_bits = 0

total number of bits (indexes + norms)

size_t norm_bits = 0

bits allocated for the norms

size_t total_codebook_size = 0

size of the codebook in vectors

bool only_8bit = false

are all nbits = 8 (use faster decoder)

bool verbose = false

verbose during training?

bool is_trained = false

is trained or not

std::vector<float> norm_tabs

auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan

IndexFlat1D qnorm

store and search norms

std::vector<float> centroid_norms

norms of all codebook entries (size total_codebook_size)

std::vector<float> codebook_cross_products

dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)

size_t max_mem_distances = 5 * (size_t(1) << 30)

norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated

Search_type_t search_type

Also determines what’s in the codes.

float norm_min = NAN

min/max for quantization of norms

float norm_max = NAN

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

Public Static Attributes

static const int Train_default = 0

regular k-means (minimal amount of computation)

static const int Train_progressive_dim = 1

progressive dim clustering (set by default)

static const int Train_refine_codebook = 2

do a few iterations of codebook refinement after first level estimation

static const int Train_top_beam = 1024

set this bit on train_type if beam is to be trained only on the first element of the beam (faster but less accurate)

static const int Skip_codebook_tables = 2048

set this bit to not autmatically compute the codebook tables after training

template<class C, bool use_sel = false>
struct BlockResultHandler

Public Functions

inline explicit BlockResultHandler(size_t nq, const IDSelector *sel = nullptr)

inline virtual void begin_multiple(size_t i0_2, size_t i1_2)

inline virtual void add_results(size_t, size_t, const typename C::T*)

inline virtual void end_multiple()

inline virtual ~BlockResultHandler()

inline bool is_in_selection(idx_t i) const

Public Members

size_t nq

const IDSelector *sel

size_t i0 = 0

size_t i1 = 0

template<class C, bool use_sel = false>
struct Top1BlockResultHandler : public faiss::BlockResultHandler<C, false>

Public Types

using T = typename C::T

using TI = typename C::TI

Public Functions

inline Top1BlockResultHandler(size_t nq, T *dis_tab, TI *ids_tab, const IDSelector *sel = nullptr)

inline virtual void begin_multiple(size_t i0, size_t i1) final

begin

inline void add_results(size_t j0, size_t j1, const T *dis_tab_2) final

add results for query i0..i1 and j0..j1

inline void add_result(const size_t i, const T dis, const TI idx)

inline virtual void add_results(size_t, size_t, const typename C::T*)

inline virtual void end_multiple()

inline bool is_in_selection(idx_t i) const

Public Members

T *dis_tab

TI *ids_tab

size_t nq

const IDSelector *sel

size_t i0

size_t i1

struct SingleResultHandler : public faiss::ResultHandler<C>

Public Functions

inline explicit SingleResultHandler(Top1BlockResultHandler &hr)

inline void begin(const size_t current_idx_2)

begin results for query # i

inline virtual bool add_result(T dis, TI idx) final

add one result for query i

inline void end()

series of results for query i is done

Public Members

Top1BlockResultHandler &hr

TI min_idx

size_t current_idx = 0

C::T threshold = C::neutral()

template<class C, bool use_sel = false>
struct HeapBlockResultHandler : public faiss::BlockResultHandler<C, false>

Public Types

using T = typename C::T

using TI = typename C::TI

Public Functions

inline HeapBlockResultHandler(size_t nq, T *heap_dis_tab, TI *heap_ids_tab, size_t k, const IDSelector *sel = nullptr)

inline virtual void begin_multiple(size_t i0_2, size_t i1_2) final

begin

inline void add_results(size_t j0, size_t j1, const T *dis_tab) final

add results for query i0..i1 and j0..j1

inline virtual void end_multiple() final

series of results for queries i0..i1 is done

inline virtual void add_results(size_t, size_t, const typename C::T*)

inline bool is_in_selection(idx_t i) const

Public Members

T *heap_dis_tab

TI *heap_ids_tab

int64_t k

size_t nq

const IDSelector *sel

size_t i0

size_t i1

struct SingleResultHandler : public faiss::ResultHandler<C>

Public Functions

inline explicit SingleResultHandler(HeapBlockResultHandler &hr)

inline void begin(size_t i)

begin results for query # i

inline virtual bool add_result(T dis, TI idx) final

add one result for query i

inline void end()

series of results for query i is done

Public Members

HeapBlockResultHandler &hr

size_t k

T *heap_dis

TI *heap_ids

C::T threshold = C::neutral()

template<class C>
struct ReservoirTopN : public faiss::ResultHandler<C>

#include <ResultHandler.h>

Reservoir for a single query.

Subclassed by faiss::ReservoirBlockResultHandler< C, use_sel >::SingleResultHandler

Public Types

using T = typename C::T

using TI = typename C::TI

Public Functions

inline ReservoirTopN()

inline ReservoirTopN(size_t n, size_t capacity, T *vals, TI *ids)

inline virtual bool add_result(T val, TI id) final

inline void add(T val, TI id)

inline void shrink_fuzzy()

inline void shrink()

inline void to_result(T *heap_dis, TI *heap_ids) const

Public Members

T *vals

TI *ids

size_t i

size_t n

size_t capacity

C::T threshold = C::neutral()

template<class C, bool use_sel = false>
struct ReservoirBlockResultHandler : public faiss::BlockResultHandler<C, false>

Public Types

using T = typename C::T

using TI = typename C::TI

Public Functions

inline ReservoirBlockResultHandler(size_t nq, T *heap_dis_tab, TI *heap_ids_tab, size_t k, const IDSelector *sel = nullptr)

inline virtual void begin_multiple(size_t i0_2, size_t i1_2)

begin

inline void add_results(size_t j0, size_t j1, const T *dis_tab)

add results for query i0..i1 and j0..j1

inline virtual void end_multiple() final

series of results for queries i0..i1 is done

inline virtual void add_results(size_t, size_t, const typename C::T*)

inline bool is_in_selection(idx_t i) const

Public Members

T *heap_dis_tab

TI *heap_ids_tab

int64_t k

size_t capacity

std::vector<T> reservoir_dis

std::vector<TI> reservoir_ids

std::vector<ReservoirTopN<C>> reservoirs

size_t nq

const IDSelector *sel

size_t i0

size_t i1

struct SingleResultHandler : public faiss::ReservoirTopN<C>

Public Types

using T = typename C::T

using TI = typename C::TI

Public Functions

inline explicit SingleResultHandler(ReservoirBlockResultHandler &hr)

inline void begin(size_t qno_2)

begin results for query # i

inline void end()

series of results for query qno is done

inline virtual bool add_result(T val, TI id) final

inline void add(T val, TI id)

inline void shrink_fuzzy()

inline void shrink()

inline void to_result(T *heap_dis, TI *heap_ids) const

Public Members

ReservoirBlockResultHandler &hr

std::vector<T> reservoir_dis

std::vector<TI> reservoir_ids

size_t qno

T *vals

TI *ids

size_t i

size_t n

size_t capacity

C::T threshold = C::neutral()

template<class C, bool use_sel = false>
struct RangeSearchBlockResultHandler : public faiss::BlockResultHandler<C, false>

Public Types

using T = typename C::T

using TI = typename C::TI

Public Functions

inline RangeSearchBlockResultHandler(RangeSearchResult *res, float radius, const IDSelector *sel = nullptr)

inline virtual void begin_multiple(size_t i0_2, size_t i1_2)

begin

inline void add_results(size_t j0, size_t j1, const T *dis_tab)

add results for query i0..i1 and j0..j1

inline ~RangeSearchBlockResultHandler()

inline virtual void add_results(size_t, size_t, const typename C::T*)

inline virtual void end_multiple()

inline bool is_in_selection(idx_t i) const

Public Members

RangeSearchResult *res

T radius

std::vector<RangeSearchPartialResult*> partial_results

std::vector<size_t> j0s

int pr = 0

size_t nq

const IDSelector *sel

size_t i0

size_t i1

struct SingleResultHandler : public faiss::ResultHandler<C>

Public Functions

inline explicit SingleResultHandler(RangeSearchBlockResultHandler &rh)

inline void begin(size_t i)

begin results for query # i

inline virtual bool add_result(T dis, TI idx) final

add one result for query i

inline void end()

series of results for query i is done

inline ~SingleResultHandler()

Public Members

RangeSearchPartialResult pres

RangeQueryResult *qr = nullptr

C::T threshold = C::neutral()

struct ScalarQuantizer : public faiss::Quantizer

#include <ScalarQuantizer.h>

The uniform quantizer has a range [vmin, vmax]. The range can be the same for all dimensions (uniform) or specific per dimension (default).

Public Types

enum QuantizerType

Values:

enumerator QT_8bit

8 bits per component

enumerator QT_4bit

4 bits per component

enumerator QT_8bit_uniform

same, shared range for all dimensions

enumerator QT_4bit_uniform

enumerator QT_fp16

enumerator QT_8bit_direct

fast indexing of uint8s

enumerator QT_6bit

6 bits per component

enumerator QT_bf16

enumerator QT_8bit_direct_signed

fast indexing of signed int8s ranging from [-128 to 127]

enum RangeStat

The uniform encoder can estimate the range of representable values of the unform encoder using different statistics. Here rs = rangestat_arg

Values:

enumerator RS_minmax

[min - rs*(max-min), max + rs*(max-min)]

enumerator RS_meanstd

[mean - std * rs, mean + std * rs]

enumerator RS_quantiles

[Q(rs), Q(1-rs)]

enumerator RS_optim

alternate optimization of reconstruction error

Public Functions

ScalarQuantizer(size_t d, QuantizerType qtype)

ScalarQuantizer()

void set_derived_sizes()

updates internal values based on qtype and d

virtual void train(size_t n, const float *x) override

Train the quantizer

Parameters:

x – training vectors, size n * d

virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override

Encode a set of vectors

Parameters:

• x – vectors to encode, size n * d

• codes – output codes, size n * code_size

virtual void decode(const uint8_t *code, float *x, size_t n) const override

Decode a set of vectors

Parameters:

• codes – codes to decode, size n * code_size

• x – output vectors, size n * d

SQuantizer *select_quantizer() const

SQDistanceComputer *get_distance_computer(MetricType metric = METRIC_L2) const

InvertedListScanner *select_InvertedListScanner(MetricType mt, const Index *quantizer, bool store_pairs, const IDSelector *sel, bool by_residual = false) const

Public Members

QuantizerType qtype = QT_8bit

RangeStat rangestat = RS_minmax

float rangestat_arg = 0

size_t bits = 0

bits per scalar code

std::vector<float> trained

trained values (including the range)

size_t d

size of the input vectors

size_t code_size

bytes per indexed vector

struct SQDistanceComputer : public faiss::FlatCodesDistanceComputer

Public Functions

inline SQDistanceComputer()

virtual float query_to_code(const uint8_t *code) const = 0

inline virtual float distance_to_code(const uint8_t *code) final

compute distance of current query to an encoded vector

inline virtual float operator()(idx_t i) override

compute distance of vector i to current query

virtual void set_query(const float *x) = 0

called before computing distances. Pointer x should remain valid while operator () is called

inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3)

compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.

virtual float symmetric_dis(idx_t i, idx_t j) = 0

compute distance between two stored vectors

Public Members

const float *q

const uint8_t *codes

size_t code_size

struct SQuantizer

Public Functions

virtual void encode_vector(const float *x, uint8_t *code) const = 0

virtual void decode_vector(const uint8_t *code, float *x) const = 0

inline virtual ~SQuantizer()

struct SIMDResultHandler

Subclassed by faiss::SIMDResultHandlerToFloat, faiss::simd_result_handlers::DummyResultHandler, faiss::simd_result_handlers::FixedStorageHandler< NQ, BB >, faiss::simd_result_handlers::StoreResultHandler

Public Functions

virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0

called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.

virtual void set_block_origin(size_t i0, size_t j0) = 0

set the sub-matrix that is being computed

inline virtual ~SIMDResultHandler()

Public Members

bool is_CMax = false

uint8_t sizeof_ids = 0

bool with_fields = false

struct SIMDResultHandlerToFloat : public faiss::SIMDResultHandler

Subclassed by faiss::simd_result_handlers::ResultHandlerCompare< C, false >, faiss::simd_result_handlers::ResultHandlerCompare< C, with_id_map >

Public Functions

inline SIMDResultHandlerToFloat(size_t nq, size_t ntotal)

inline virtual void begin(const float *norms)

inline virtual void end()

virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0

called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.

virtual void set_block_origin(size_t i0, size_t j0) = 0

set the sub-matrix that is being computed

Public Members

size_t nq

size_t ntotal

const idx_t *id_map = nullptr

these fields are used mainly for the IVF variants (with_id_map=true)

const int *q_map = nullptr

const uint16_t *dbias = nullptr

const float *normalizers = nullptr

bool is_CMax = false

uint8_t sizeof_ids = 0

bool with_fields = false

template<typename IndexT>
class ThreadedIndex : public IndexT

#include <ThreadedIndex.h>

A holder of indices in a collection of threads The interface to this class itself is not thread safe

Subclassed by faiss::IndexReplicasTemplate< IndexT >, faiss::IndexShardsTemplate< IndexT >

Public Functions

explicit ThreadedIndex(bool threaded)

explicit ThreadedIndex(int d, bool threaded)

~ThreadedIndex() override

virtual void addIndex(IndexT *index)

override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.

void removeIndex(IndexT *index)

Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.

void runOnIndex(std::function<void(int, IndexT*)> f)

Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)

void runOnIndex(std::function<void(int, const IndexT*)> f) const

void reset() override

faiss::Index API All indices receive the same call

inline int count() const

Returns the number of sub-indices.

inline IndexT *at(size_t i)

Returns the i-th sub-index.

inline const IndexT *at(size_t i) const

Returns the i-th sub-index (const version)

Public Members

bool own_indices = false

Whether or not we are responsible for deleting our contained indices.

Protected Functions

virtual void onAfterAddIndex(IndexT *index)

Called just after an index is added.

virtual void onAfterRemoveIndex(IndexT *index)

Called just after an index is removed.

Protected Attributes

std::vector<std::pair<IndexT*, std::unique_ptr<WorkerThread>>> indices_

Collection of Index instances, with their managing worker thread if any.

bool isThreaded_

Is this index multi-threaded?

Protected Static Functions

static void waitAndHandleFutures(std::vector<std::future<bool>> &v)

struct SearchParameters

#include <Index.h>

Parent class for the optional search paramenters.

Sub-classes with additional search parameters should inherit this class. Ownership of the object fields is always to the caller.

Subclassed by faiss::IndexRefineSearchParameters, faiss::SearchParametersHNSW, faiss::SearchParametersIVF, faiss::SearchParametersPQ, faiss::SearchParametersPreTransform, faiss::SearchParametersResidualCoarseQuantizer, faiss::gpu::SearchParametersCagra

Public Functions

inline virtual ~SearchParameters()

make sure we can dynamic_cast this

Public Members

IDSelector *sel = nullptr

if non-null, only these IDs will be considered during search.

struct Index

#include <Index.h>

Abstract structure for an index, supports adding vectors and searching them.

All vectors provided at add or search time are 32-bit float arrays, although the internal representation may vary.

Subclassed by faiss::AdditiveCoarseQuantizer, faiss::IndexFastScan, faiss::IndexFlatCodes, faiss::IndexHNSW, faiss::IndexIVF, faiss::IndexIVFIndependentQuantizer, faiss::IndexNNDescent, faiss::IndexNSG, faiss::IndexPreTransform, faiss::IndexRandom, faiss::IndexRefine, faiss::IndexRowwiseMinMaxBase, faiss::IndexSplitVectors, faiss::MultiIndexQuantizer, faiss::gpu::GpuIndex

Public Types

using component_t = float

using distance_t = float

Public Functions

inline explicit Index(idx_t d = 0, MetricType metric = METRIC_L2)

virtual ~Index()

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add(idx_t n, const float *x) = 0

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const = 0

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reset() = 0

removes all elements from the database.

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct(idx_t key, float *recons) const

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct Index2Layer : public faiss::IndexFlatCodes

#include <Index2Layer.h>

Same as an IndexIVFPQ without the inverted lists: codes are stored sequentially

The class is mainly inteded to store encoded vectors that can be accessed randomly, the search function is not implemented.

Public Types

using component_t = float

using distance_t = float

Public Functions

Index2Layer(Index *quantizer, size_t nlist, int M, int nbit = 8, MetricType metric = METRIC_L2)

Index2Layer()

~Index2Layer()

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

not implemented

virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

void transfer_to_IVFPQ(IndexIVFPQ &other) const

transfer the flat codes to an IVFPQ index

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

Level1Quantizer q1

first level quantizer

ProductQuantizer pq

second level quantizer is always a PQ

size_t code_size_1

size of the code for the first level (ceil(log8(q1.nlist)))

size_t code_size_2

size of the code for the second level

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexAdditiveQuantizer : public faiss::IndexFlatCodes

#include <IndexAdditiveQuantizer.h>

Abstract class for additive quantizers. The search functions are in common.

Subclassed by faiss::IndexLocalSearchQuantizer, faiss::IndexProductLocalSearchQuantizer, faiss::IndexProductResidualQuantizer, faiss::IndexResidualQuantizer

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

explicit IndexAdditiveQuantizer(idx_t d, AdditiveQuantizer *aq, MetricType metric = METRIC_L2)

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

AdditiveQuantizer *aq

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexResidualQuantizer : public faiss::IndexAdditiveQuantizer

#include <IndexAdditiveQuantizer.h>

Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexResidualQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

IndexResidualQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

IndexResidualQuantizer()

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

ResidualQuantizer rq

The residual quantizer used to encode the vectors.

AdditiveQuantizer *aq

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexLocalSearchQuantizer : public faiss::IndexAdditiveQuantizer

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexLocalSearchQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

IndexLocalSearchQuantizer()

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

LocalSearchQuantizer lsq

AdditiveQuantizer *aq

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexProductResidualQuantizer : public faiss::IndexAdditiveQuantizer

#include <IndexAdditiveQuantizer.h>

Index based on a product residual quantizer.

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexProductResidualQuantizer(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• nsplits – number of residual quantizers

• Msub – number of subquantizers per RQ

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• nsplits – number of residual quantizers

• Msub – number of subquantizers per RQ

• nbits – number of bit per subvector index

IndexProductResidualQuantizer()

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

ProductResidualQuantizer prq

The product residual quantizer used to encode the vectors.

AdditiveQuantizer *aq

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexProductLocalSearchQuantizer : public faiss::IndexAdditiveQuantizer

#include <IndexAdditiveQuantizer.h>

Index based on a product local search quantizer.

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexProductLocalSearchQuantizer(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• nsplits – number of local search quantizers

• Msub – number of subquantizers per LSQ

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• nsplits – number of local search quantizers

• Msub – number of subquantizers per LSQ

• nbits – number of bit per subvector index

IndexProductLocalSearchQuantizer()

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

ProductLocalSearchQuantizer plsq

The product local search quantizer used to encode the vectors.

AdditiveQuantizer *aq

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct AdditiveCoarseQuantizer : public faiss::Index

#include <IndexAdditiveQuantizer.h>

A “virtual” index where the elements are the residual quantizer centroids.

Intended for use as a coarse quantizer in an IndexIVF.

Subclassed by faiss::LocalSearchCoarseQuantizer, faiss::ResidualCoarseQuantizer

Public Types

using component_t = float

using distance_t = float

Public Functions

explicit AdditiveCoarseQuantizer(idx_t d = 0, AdditiveQuantizer *aq = nullptr, MetricType metric = METRIC_L2)

virtual void add(idx_t n, const float *x) override

N/A.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void reset() override

N/A.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

AdditiveQuantizer *aq

std::vector<float> centroid_norms

norms of centroids, useful for knn-search

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct SearchParametersResidualCoarseQuantizer : public faiss::SearchParameters

Public Functions

inline ~SearchParametersResidualCoarseQuantizer()

Public Members

float beam_factor = 4.0f

IDSelector *sel = nullptr

if non-null, only these IDs will be considered during search.

struct ResidualCoarseQuantizer : public faiss::AdditiveCoarseQuantizer

#include <IndexAdditiveQuantizer.h>

The ResidualCoarseQuantizer is a bit specialized compared to the default AdditiveCoarseQuantizer because it can use a beam search at search time (slow but may be useful for very large vocabularies)

Public Types

using component_t = float

using distance_t = float

Public Functions

void set_beam_factor(float new_beam_factor)

computes centroid norms if required

ResidualCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

ResidualCoarseQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2)

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

void initialize_from(const ResidualCoarseQuantizer &other)

Copy the M first codebook levels from other. Useful to crop a ResidualQuantizer to its first M quantizers.

ResidualCoarseQuantizer()

virtual void add(idx_t n, const float *x) override

N/A.

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void reset() override

N/A.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

ResidualQuantizer rq

The residual quantizer used to encode the vectors.

float beam_factor = 4.0f

factor between the beam size and the search k if negative, use exact search-to-centroid

AdditiveQuantizer *aq

std::vector<float> centroid_norms

norms of centroids, useful for knn-search

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct LocalSearchCoarseQuantizer : public faiss::AdditiveCoarseQuantizer

Public Types

using component_t = float

using distance_t = float

Public Functions

LocalSearchCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

LocalSearchCoarseQuantizer()

virtual void add(idx_t n, const float *x) override

N/A.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void reset() override

N/A.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

LocalSearchQuantizer lsq

The residual quantizer used to encode the vectors.

AdditiveQuantizer *aq

std::vector<float> centroid_norms

norms of centroids, useful for knn-search

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexAdditiveQuantizerFastScan : public faiss::IndexFastScan

#include <IndexAdditiveQuantizerFastScan.h>

Fast scan version of IndexAQ. Works for 4-bit AQ for now.

The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions.

Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator

Subclassed by faiss::IndexLocalSearchQuantizerFastScan, faiss::IndexProductLocalSearchQuantizerFastScan, faiss::IndexProductResidualQuantizerFastScan, faiss::IndexResidualQuantizerFastScan

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

explicit IndexAdditiveQuantizerFastScan(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)

void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)

IndexAdditiveQuantizerFastScan()

~IndexAdditiveQuantizerFastScan() override

explicit IndexAdditiveQuantizerFastScan(const IndexAdditiveQuantizer &orig, int bbs = 32)

build from an existing IndexAQ

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

void estimate_norm_scale(idx_t n, const float *x)

virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override

virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

Decode a set of vectors.

NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * code_size

• x – output vectors, size n * d

void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)

virtual void reset() override

removes all elements from the database.

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const

template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t remove_ids(const IDSelector &sel) override

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

CodePacker *get_CodePacker() const

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

AdditiveQuantizer *aq

bool rescale_norm = true

int norm_scale = 1

size_t max_train_points = 0

int implem = 0

int skip = 0

int bbs

int qbs = 0

size_t M

size_t nbits

size_t ksub

size_t code_size

size_t ntotal2

size_t M2

AlignedTable<uint8_t> codes

const uint8_t *orig_codes = nullptr

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexResidualQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan

#include <IndexAdditiveQuantizerFastScan.h>

Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexResidualQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• metric – metric type

• search_type – AQ search type

IndexResidualQuantizerFastScan()

void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

void estimate_norm_scale(idx_t n, const float *x)

virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override

virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

Decode a set of vectors.

NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * code_size

• x – output vectors, size n * d

void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)

virtual void reset() override

removes all elements from the database.

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const

template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t remove_ids(const IDSelector &sel) override

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

CodePacker *get_CodePacker() const

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

ResidualQuantizer rq

The residual quantizer used to encode the vectors.

AdditiveQuantizer *aq

bool rescale_norm = true

int norm_scale = 1

size_t max_train_points = 0

int implem = 0

int skip = 0

int bbs

int qbs = 0

size_t M

size_t nbits

size_t ksub

size_t code_size

size_t ntotal2

size_t M2

AlignedTable<uint8_t> codes

const uint8_t *orig_codes = nullptr

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexLocalSearchQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan

#include <IndexAdditiveQuantizerFastScan.h>

Index based on a local search quantizer. Stored vectors are approximated by local search quantization codes. Can also be used as a codec

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexLocalSearchQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

• metric – metric type

• search_type – AQ search type

IndexLocalSearchQuantizerFastScan()

void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

void estimate_norm_scale(idx_t n, const float *x)

virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override

virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

Decode a set of vectors.

NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * code_size

• x – output vectors, size n * d

void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)

virtual void reset() override

removes all elements from the database.

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const

template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t remove_ids(const IDSelector &sel) override

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

CodePacker *get_CodePacker() const

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

LocalSearchQuantizer lsq

AdditiveQuantizer *aq

bool rescale_norm = true

int norm_scale = 1

size_t max_train_points = 0

int implem = 0

int skip = 0

int bbs

int qbs = 0

size_t M

size_t nbits

size_t ksub

size_t code_size

size_t ntotal2

size_t M2

AlignedTable<uint8_t> codes

const uint8_t *orig_codes = nullptr

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexProductResidualQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan

#include <IndexAdditiveQuantizerFastScan.h>

Index based on a product residual quantizer. Stored vectors are approximated by product residual quantization codes. Can also be used as a codec

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexProductResidualQuantizerFastScan(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• nsplits – number of residual quantizers

• Msub – number of subquantizers per RQ

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• nsplits – number of residual quantizers

• Msub – number of subquantizers per RQ

• nbits – number of bit per subvector index

• metric – metric type

• search_type – AQ search type

IndexProductResidualQuantizerFastScan()

void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

void estimate_norm_scale(idx_t n, const float *x)

virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override

virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

Decode a set of vectors.

NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * code_size

• x – output vectors, size n * d

void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)

virtual void reset() override

removes all elements from the database.

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const

template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t remove_ids(const IDSelector &sel) override

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

CodePacker *get_CodePacker() const

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

ProductResidualQuantizer prq

The product residual quantizer used to encode the vectors.

AdditiveQuantizer *aq

bool rescale_norm = true

int norm_scale = 1

size_t max_train_points = 0

int implem = 0

int skip = 0

int bbs

int qbs = 0

size_t M

size_t nbits

size_t ksub

size_t code_size

size_t ntotal2

size_t M2

AlignedTable<uint8_t> codes

const uint8_t *orig_codes = nullptr

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexProductLocalSearchQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan

#include <IndexAdditiveQuantizerFastScan.h>

Index based on a product local search quantizer. Stored vectors are approximated by product local search quantization codes. Can also be used as a codec

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexProductLocalSearchQuantizerFastScan(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• nsplits – number of local search quantizers

• Msub – number of subquantizers per LSQ

• nbits – number of bit per subvector index

• d – dimensionality of the input vectors

• nsplits – number of local search quantizers

• Msub – number of subquantizers per LSQ

• nbits – number of bit per subvector index

• metric – metric type

• search_type – AQ search type

IndexProductLocalSearchQuantizerFastScan()

void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)

virtual void train(idx_t n, const float *x) override

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

void estimate_norm_scale(idx_t n, const float *x)

virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override

virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

Decode a set of vectors.

NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * code_size

• x – output vectors, size n * d

void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)

virtual void reset() override

removes all elements from the database.

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const

template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t remove_ids(const IDSelector &sel) override

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

CodePacker *get_CodePacker() const

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

ProductLocalSearchQuantizer plsq

The product local search quantizer used to encode the vectors.

AdditiveQuantizer *aq

bool rescale_norm = true

int norm_scale = 1

size_t max_train_points = 0

int implem = 0

int skip = 0

int bbs

int qbs = 0

size_t M

size_t nbits

size_t ksub

size_t code_size

size_t ntotal2

size_t M2

AlignedTable<uint8_t> codes

const uint8_t *orig_codes = nullptr

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexBinary

#include <IndexBinary.h>

Abstract structure for a binary index.

Supports adding vertices and searching them.

All queries are symmetric because there is no distinction between codes and vectors.

Subclassed by faiss::IndexBinaryFlat, faiss::IndexBinaryFromFloat, faiss::IndexBinaryHNSW, faiss::IndexBinaryHash, faiss::IndexBinaryIVF, faiss::IndexBinaryMultiHash, faiss::gpu::GpuIndexBinaryFlat

Public Types

using component_t = uint8_t

using distance_t = int32_t

Public Functions

explicit IndexBinary(idx_t d = 0, MetricType metric = METRIC_L2)

virtual ~IndexBinary()

virtual void train(idx_t n, const uint8_t *x)

Perform training on a representative set of vectors.

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d / 8

virtual void add(idx_t n, const uint8_t *x) = 0

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1

Parameters:

x – input matrix, size n * d / 8

virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

xids – if non-null, ids to store for the vectors (size n)

virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const = 0

Query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

• distances – output pairwise distances, size n*k

virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

Query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.

Parameters:

• x – input vectors to search, size n * d / 8

• radius – search radius

• result – result table

void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const

Return the indexes of the k vectors closest to the query x.

This function is identical to search but only returns labels of neighbors.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

virtual void reset() = 0

Removes all elements from the database.

virtual size_t remove_ids(const IDSelector &sel)

Removes IDs from the index. Not supported by all indexes.

virtual void reconstruct(idx_t key, uint8_t *recons) const

Reconstruct a stored vector.

This function may not be defined for some indexes.

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d / 8)

virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const

Reconstruct vectors i0 to i0 + ni - 1.

This function may not be defined for some indexes.

Parameters:

recons – reconstucted vectors (size ni * d / 8)

virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting array is padded with -1s.

Parameters:

recons – reconstructed vectors size (n, k, d)

void display() const

Display the actual class name and some more info.

virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Same as add_with_ids for IndexBinary.

Public Members

int d = 0

vector dimension

int code_size = 0

number of bytes per vector ( = d / 8 )

idx_t ntotal = 0

total nb of indexed vectors

bool verbose = false

verbosity level

bool is_trained = true

set if the Index does not require training, or if training is done already

MetricType metric_type = METRIC_L2

type of metric this index uses for search

struct IndexBinaryFlat : public faiss::IndexBinary

#include <IndexBinaryFlat.h>

Index that stores the full vectors and performs exhaustive search.

Public Types

using component_t = uint8_t

using distance_t = int32_t

Public Functions

explicit IndexBinaryFlat(idx_t d)

virtual void add(idx_t n, const uint8_t *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1

Parameters:

x – input matrix, size n * d / 8

virtual void reset() override

Removes all elements from the database.

virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

• distances – output pairwise distances, size n*k

virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.

Parameters:

• x – input vectors to search, size n * d / 8

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, uint8_t *recons) const override

Reconstruct a stored vector.

This function may not be defined for some indexes.

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d / 8)

virtual size_t remove_ids(const IDSelector &sel) override

Remove some ids. Note that because of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted.

inline IndexBinaryFlat()

virtual void train(idx_t n, const uint8_t *x)

Perform training on a representative set of vectors.

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d / 8

virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

xids – if non-null, ids to store for the vectors (size n)

void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const

Return the indexes of the k vectors closest to the query x.

This function is identical to search but only returns labels of neighbors.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const

Reconstruct vectors i0 to i0 + ni - 1.

This function may not be defined for some indexes.

Parameters:

recons – reconstucted vectors (size ni * d / 8)

virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting array is padded with -1s.

Parameters:

recons – reconstructed vectors size (n, k, d)

void display() const

Display the actual class name and some more info.

virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Same as add_with_ids for IndexBinary.

Public Members

std::vector<uint8_t> xb

database vectors, size ntotal * d / 8

bool use_heap = true

Select between using a heap or counting to select the k smallest values when scanning inverted lists.

size_t query_batch_size = 32

ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK

int d = 0

vector dimension

int code_size = 0

number of bytes per vector ( = d / 8 )

idx_t ntotal = 0

total nb of indexed vectors

bool verbose = false

verbosity level

bool is_trained = true

set if the Index does not require training, or if training is done already

MetricType metric_type = METRIC_L2

type of metric this index uses for search

struct IndexBinaryFromFloat : public faiss::IndexBinary

#include <IndexBinaryFromFloat.h>

IndexBinary backed by a float Index.

Supports adding vertices and searching them.

All queries are symmetric because there is no distinction between codes and vectors.

Public Types

using component_t = uint8_t

using distance_t = int32_t

Public Functions

IndexBinaryFromFloat()

explicit IndexBinaryFromFloat(Index *index)

~IndexBinaryFromFloat()

virtual void add(idx_t n, const uint8_t *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1

Parameters:

x – input matrix, size n * d / 8

virtual void reset() override

Removes all elements from the database.

virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

• distances – output pairwise distances, size n*k

virtual void train(idx_t n, const uint8_t *x) override

Perform training on a representative set of vectors.

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d / 8

virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

Query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.

Parameters:

• x – input vectors to search, size n * d / 8

• radius – search radius

• result – result table

void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const

Return the indexes of the k vectors closest to the query x.

This function is identical to search but only returns labels of neighbors.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

virtual size_t remove_ids(const IDSelector &sel)

Removes IDs from the index. Not supported by all indexes.

virtual void reconstruct(idx_t key, uint8_t *recons) const

Reconstruct a stored vector.

This function may not be defined for some indexes.

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d / 8)

virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const

Reconstruct vectors i0 to i0 + ni - 1.

This function may not be defined for some indexes.

Parameters:

recons – reconstucted vectors (size ni * d / 8)

virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting array is padded with -1s.

Parameters:

recons – reconstructed vectors size (n, k, d)

void display() const

Display the actual class name and some more info.

virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Same as add_with_ids for IndexBinary.

Public Members

Index *index = nullptr

bool own_fields = false

Whether object owns the index pointer.

int d = 0

vector dimension

int code_size = 0

number of bytes per vector ( = d / 8 )

idx_t ntotal = 0

total nb of indexed vectors

bool verbose = false

verbosity level

bool is_trained = true

set if the Index does not require training, or if training is done already

MetricType metric_type = METRIC_L2

type of metric this index uses for search

struct IndexBinaryHash : public faiss::IndexBinary

#include <IndexBinaryHash.h>

just uses the b first bits as a hash value

Public Types

using InvertedListMap = std::unordered_map<idx_t, InvertedList>

using component_t = uint8_t

using distance_t = int32_t

Public Functions

IndexBinaryHash(int d, int b)

IndexBinaryHash()

virtual void reset() override

Removes all elements from the database.

virtual void add(idx_t n, const uint8_t *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1

Parameters:

x – input matrix, size n * d / 8

virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.

Parameters:

• x – input vectors to search, size n * d / 8

• radius – search radius

• result – result table

virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

• distances – output pairwise distances, size n*k

void display() const

size_t hashtable_size() const

virtual void train(idx_t n, const uint8_t *x)

Perform training on a representative set of vectors.

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d / 8

void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const

Return the indexes of the k vectors closest to the query x.

This function is identical to search but only returns labels of neighbors.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

virtual size_t remove_ids(const IDSelector &sel)

Removes IDs from the index. Not supported by all indexes.

virtual void reconstruct(idx_t key, uint8_t *recons) const

Reconstruct a stored vector.

This function may not be defined for some indexes.

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d / 8)

virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const

Reconstruct vectors i0 to i0 + ni - 1.

This function may not be defined for some indexes.

Parameters:

recons – reconstucted vectors (size ni * d / 8)

virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting array is padded with -1s.

Parameters:

recons – reconstructed vectors size (n, k, d)

virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Same as add_with_ids for IndexBinary.

Public Members

InvertedListMap invlists

int b

int nflip

int d = 0

vector dimension

int code_size = 0

number of bytes per vector ( = d / 8 )

idx_t ntotal = 0

total nb of indexed vectors

bool verbose = false

verbosity level

bool is_trained = true

set if the Index does not require training, or if training is done already

MetricType metric_type = METRIC_L2

type of metric this index uses for search

struct InvertedList

Public Functions

void add(idx_t id, size_t code_size, const uint8_t *code)

Public Members

std::vector<idx_t> ids

std::vector<uint8_t> vecs

struct IndexBinaryHashStats

Public Functions

inline IndexBinaryHashStats()

void reset()

Public Members

size_t nq

size_t n0

size_t nlist

size_t ndis

struct IndexBinaryMultiHash : public faiss::IndexBinary

#include <IndexBinaryHash.h>

just uses the b first bits as a hash value

Public Types

using Map = std::unordered_map<idx_t, std::vector<idx_t>>

using component_t = uint8_t

using distance_t = int32_t

Public Functions

IndexBinaryMultiHash(int d, int nhash, int b)

IndexBinaryMultiHash()

~IndexBinaryMultiHash()

virtual void reset() override

Removes all elements from the database.

virtual void add(idx_t n, const uint8_t *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1

Parameters:

x – input matrix, size n * d / 8

virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.

Parameters:

• x – input vectors to search, size n * d / 8

• radius – search radius

• result – result table

virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

• distances – output pairwise distances, size n*k

size_t hashtable_size() const

virtual void train(idx_t n, const uint8_t *x)

Perform training on a representative set of vectors.

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d / 8

virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

xids – if non-null, ids to store for the vectors (size n)

void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const

Return the indexes of the k vectors closest to the query x.

This function is identical to search but only returns labels of neighbors.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

virtual size_t remove_ids(const IDSelector &sel)

Removes IDs from the index. Not supported by all indexes.

virtual void reconstruct(idx_t key, uint8_t *recons) const

Reconstruct a stored vector.

This function may not be defined for some indexes.

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d / 8)

virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const

Reconstruct vectors i0 to i0 + ni - 1.

This function may not be defined for some indexes.

Parameters:

recons – reconstucted vectors (size ni * d / 8)

virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting array is padded with -1s.

Parameters:

recons – reconstructed vectors size (n, k, d)

void display() const

Display the actual class name and some more info.

virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Same as add_with_ids for IndexBinary.

Public Members

IndexBinaryFlat *storage

bool own_fields

std::vector<Map> maps

int nhash

nb of hash maps

int b

nb bits per hash map

int nflip

nb bit flips to use at search time

int d = 0

vector dimension

int code_size = 0

number of bytes per vector ( = d / 8 )

idx_t ntotal = 0

total nb of indexed vectors

bool verbose = false

verbosity level

bool is_trained = true

set if the Index does not require training, or if training is done already

MetricType metric_type = METRIC_L2

type of metric this index uses for search

struct IndexBinaryHNSW : public faiss::IndexBinary

#include <IndexBinaryHNSW.h>

The HNSW index is a normal random-access index with a HNSW link structure built on top

Public Types

typedef HNSW::storage_idx_t storage_idx_t

using component_t = uint8_t

using distance_t = int32_t

Public Functions

explicit IndexBinaryHNSW()

explicit IndexBinaryHNSW(int d, int M = 32)

explicit IndexBinaryHNSW(IndexBinary *storage, int M = 32)

~IndexBinaryHNSW() override

DistanceComputer *get_distance_computer() const

virtual void add(idx_t n, const uint8_t *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1

Parameters:

x – input matrix, size n * d / 8

virtual void train(idx_t n, const uint8_t *x) override

Trains the storage if needed.

virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

entry point for search

virtual void reconstruct(idx_t key, uint8_t *recons) const override

Reconstruct a stored vector.

This function may not be defined for some indexes.

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d / 8)

virtual void reset() override

Removes all elements from the database.

virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

Query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.

Parameters:

• x – input vectors to search, size n * d / 8

• radius – search radius

• result – result table

void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const

Return the indexes of the k vectors closest to the query x.

This function is identical to search but only returns labels of neighbors.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

virtual size_t remove_ids(const IDSelector &sel)

Removes IDs from the index. Not supported by all indexes.

virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const

Reconstruct vectors i0 to i0 + ni - 1.

This function may not be defined for some indexes.

Parameters:

recons – reconstucted vectors (size ni * d / 8)

virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting array is padded with -1s.

Parameters:

recons – reconstructed vectors size (n, k, d)

void display() const

Display the actual class name and some more info.

virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Same as add_with_ids for IndexBinary.

Public Members

HNSW hnsw

bool own_fields

IndexBinary *storage

int d = 0

vector dimension

int code_size = 0

number of bytes per vector ( = d / 8 )

idx_t ntotal = 0

total nb of indexed vectors

bool verbose = false

verbosity level

bool is_trained = true

set if the Index does not require training, or if training is done already

MetricType metric_type = METRIC_L2

type of metric this index uses for search

struct IndexBinaryIVF : public faiss::IndexBinary

#include <IndexBinaryIVF.h>

Index based on a inverted file (IVF)

In the inverted file, the quantizer (an IndexBinary instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.

Otherwise the object is similar to the IndexIVF

Public Types

using component_t = uint8_t

using distance_t = int32_t

Public Functions

IndexBinaryIVF(IndexBinary *quantizer, size_t d, size_t nlist)

The Inverted file takes a quantizer (an IndexBinary) on input, which implements the function mapping a vector to a list identifier. The pointer is borrowed: the quantizer should not be deleted while the IndexBinaryIVF is in use.

IndexBinaryIVF()

~IndexBinaryIVF() override

virtual void reset() override

Removes all elements from the database.

virtual void train(idx_t n, const uint8_t *x) override

Trains the quantizer.

virtual void add(idx_t n, const uint8_t *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1

Parameters:

x – input matrix, size n * d / 8

virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

xids – if non-null, ids to store for the vectors (size n)

void add_core(idx_t n, const uint8_t *x, const idx_t *xids, const idx_t *precomputed_idx)

Implementation of vector addition where the vector assignments are predefined.

Parameters:

precomputed_idx – quantization indices for the input vectors (size n)

void search_preassigned(idx_t n, const uint8_t *x, idx_t k, const idx_t *assign, const int32_t *centroid_dis, int32_t *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr) const

Search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. search() calls this.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• distance – output distances, size n * k

• labels – output labels, size n * k

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

virtual BinaryInvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const

virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

assign the vectors, then call search_preassign

virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

Query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.

Parameters:

• x – input vectors to search, size n * d / 8

• radius – search radius

• result – result table

void range_search_preassigned(idx_t n, const uint8_t *x, int radius, const idx_t *assign, const int32_t *centroid_dis, RangeSearchResult *result) const

virtual void reconstruct(idx_t key, uint8_t *recons) const override

Reconstruct a stored vector.

This function may not be defined for some indexes.

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d / 8)

virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const override

Reconstruct a subset of the indexed vectors.

Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.

Parameters:

• i0 – first vector to reconstruct

• ni – nb of vectors to reconstruct

• recons – output array of reconstructed vectors, size ni * d / 8

virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const override

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the store_pairs flag in search_preassigned().

Parameters:

recons – reconstructed vectors size (n, k, d / 8)

virtual void reconstruct_from_offset(idx_t list_no, idx_t offset, uint8_t *recons) const

Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.

Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with store_pairs set.

virtual size_t remove_ids(const IDSelector &sel) override

Dataset manipulation functions.

virtual void merge_from(IndexBinary &other, idx_t add_id) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

inline size_t get_list_size(size_t list_no) const

void make_direct_map(bool new_maintain_direct_map = true)

initialize a direct map

Parameters:

new_maintain_direct_map – if true, create a direct map, else clear it

void set_direct_map_type(DirectMap::Type type)

void replace_invlists(InvertedLists *il, bool own = false)

void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const

Return the indexes of the k vectors closest to the query x.

This function is identical to search but only returns labels of neighbors.

Parameters:

• x – input vectors to search, size n * d / 8

• labels – output labels of the NNs, size n*k

void display() const

Display the actual class name and some more info.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Same as add_with_ids for IndexBinary.

Public Members

InvertedLists *invlists = nullptr

Access to the actual data.

bool own_invlists = true

size_t nprobe = 1

number of probes at query time

size_t max_codes = 0

max nb of codes to visit to do a query

bool use_heap = true

Select between using a heap or counting to select the k smallest values when scanning inverted lists.

bool per_invlist_search = false

collect computations per batch

DirectMap direct_map

map for direct access to the elements. Enables reconstruct().

IndexBinary *quantizer = nullptr

quantizer that maps vectors to inverted lists

size_t nlist = 0

number of possible key values

bool own_fields = false

whether object owns the quantizer

ClusteringParameters cp

to override default clustering params

Index *clustering_index = nullptr

to override index used during clustering

int d = 0

vector dimension

int code_size = 0

number of bytes per vector ( = d / 8 )

idx_t ntotal = 0

total nb of indexed vectors

bool verbose = false

verbosity level

bool is_trained = true

set if the Index does not require training, or if training is done already

MetricType metric_type = METRIC_L2

type of metric this index uses for search

struct BinaryInvertedListScanner

Public Functions

virtual void set_query(const uint8_t *query_vector) = 0

from now on we handle this query.

virtual void set_list(idx_t list_no, uint8_t coarse_dis) = 0

following codes come from this inverted list

virtual uint32_t distance_to_code(const uint8_t *code) const = 0

compute a single query-to-code distance

virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, int32_t *distances, idx_t *labels, size_t k) const = 0

compute the distances to codes. (distances, labels) should be organized as a min- or max-heap

Parameters:

• n – number of codes to scan

• codes – codes to scan (n * code_size)

• ids – corresponding ids (ignored if store_pairs)

• distances – heap distances (size k)

• labels – heap labels (size k)

• k – heap size

virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, int radius, RangeQueryResult &result) const = 0

inline virtual ~BinaryInvertedListScanner()

struct IndexFastScan : public faiss::Index

#include <IndexFastScan.h>

Fast scan version of IndexPQ and IndexAQ. Works for 4-bit PQ and AQ for now.

The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions. The trailing codes (padding codes that are added to complete the last code) are garbage.

Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator

Subclassed by faiss::IndexAdditiveQuantizerFastScan, faiss::IndexPQFastScan

Public Types

using component_t = float

using distance_t = float

Public Functions

void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)

IndexFastScan()

virtual void reset() override

removes all elements from the database.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const = 0

virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const = 0

void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const

template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const

template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t remove_ids(const IDSelector &sel) override

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

CodePacker *get_CodePacker() const

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

int implem = 0

int skip = 0

int bbs

int qbs = 0

size_t M

size_t nbits

size_t ksub

size_t code_size

size_t ntotal2

size_t M2

AlignedTable<uint8_t> codes

const uint8_t *orig_codes = nullptr

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct FastScanStats

Public Functions

inline FastScanStats()

inline void reset()

Public Members

uint64_t t0

uint64_t t1

uint64_t t2

uint64_t t3

struct IndexFlat : public faiss::IndexFlatCodes

#include <IndexFlat.h>

Index that stores the full vectors and performs exhaustive search

Subclassed by faiss::IndexFlatIP, faiss::IndexFlatL2

Public Types

using component_t = float

using distance_t = float

Public Functions

explicit IndexFlat(idx_t d, MetricType metric = METRIC_L2)

Parameters:

d – dimensionality of the input vectors

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const

compute distance with a subset of vectors

Parameters:

• x – query vectors, size n * d

• labels – indices of the vectors that should be compared for each query vector, size n * k

• distances – corresponding output distances, size n * k

inline float *get_xb()

inline const float *get_xb() const

inline IndexFlat()

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexFlatIP : public faiss::IndexFlat

Public Types

using component_t = float

using distance_t = float

Public Functions

inline explicit IndexFlatIP(idx_t d)

inline IndexFlatIP()

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const

compute distance with a subset of vectors

Parameters:

• x – query vectors, size n * d

• labels – indices of the vectors that should be compared for each query vector, size n * k

• distances – corresponding output distances, size n * k

inline float *get_xb()

inline const float *get_xb() const

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexFlatL2 : public faiss::IndexFlat

Subclassed by faiss::IndexFlat1D

Public Types

using component_t = float

using distance_t = float

Public Functions

inline explicit IndexFlatL2(idx_t d)

Parameters:

d – dimensionality of the input vectors

inline IndexFlatL2()

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

void sync_l2norms()

void clear_l2norms()

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const

compute distance with a subset of vectors

Parameters:

• x – query vectors, size n * d

• labels – indices of the vectors that should be compared for each query vector, size n * k

• distances – corresponding output distances, size n * k

inline float *get_xb()

inline const float *get_xb() const

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

std::vector<float> cached_l2norms

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexFlat1D : public faiss::IndexFlatL2

#include <IndexFlat.h>

optimized version for 1D “vectors”.

Public Types

using component_t = float

using distance_t = float

Public Functions

explicit IndexFlat1D(bool continuous_update = true)

void update_permutation()

if not continuous_update, call this between the last add and the first search

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void reset() override

removes all elements from the database.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

Warn: the distances returned are L1 not L2.

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

void sync_l2norms()

void clear_l2norms()

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const

compute distance with a subset of vectors

Parameters:

• x – query vectors, size n * d

• labels – indices of the vectors that should be compared for each query vector, size n * k

• distances – corresponding output distances, size n * k

inline float *get_xb()

inline const float *get_xb() const

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

Public Members

bool continuous_update = true

is the permutation updated continuously?

std::vector<idx_t> perm

sorted database indices

std::vector<float> cached_l2norms

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexFlatCodes : public faiss::Index

#include <IndexFlatCodes.h>

Index that encodes all vectors as fixed-size codes (size code_size). Storage is in the codes vector

Subclassed by faiss::Index2Layer, faiss::IndexAdditiveQuantizer, faiss::IndexFlat, faiss::IndexLSH, faiss::IndexLattice, faiss::IndexNeuralNetCodec, faiss::IndexPQ, faiss::IndexScalarQuantizer

Public Types

using component_t = float

using distance_t = float

Public Functions

IndexFlatCodes()

IndexFlatCodes(size_t code_size, idx_t d, MetricType metric = METRIC_L2)

virtual void add(idx_t n, const float *x) override

default add uses sa_encode

virtual void reset() override

removes all elements from the database.

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual size_t remove_ids(const IDSelector &sel) override

remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted

virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const

a FlatCodesDistanceComputer offers a distance_to_code method

The default implementation explicitly decodes the vector with sa_decode.

inline virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

Search implemented by decoding

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

CodePacker *get_CodePacker() const

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

void permute_entries(const idx_t *perm)

virtual void train(idx_t n, const float *x)

Perform training on a representative set of vectors

Parameters:

• n – nb of training vectors

• x – training vecors, size n * d

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

Public Members

size_t code_size

std::vector<uint8_t> codes

encoded dataset, size ntotal * code_size

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexHNSW : public faiss::Index

#include <IndexHNSW.h>

The HNSW index is a normal random-access index with a HNSW link structure built on top

Subclassed by faiss::IndexHNSW2Level, faiss::IndexHNSWCagra, faiss::IndexHNSWFlat, faiss::IndexHNSWPQ, faiss::IndexHNSWSQ

Public Types

typedef HNSW::storage_idx_t storage_idx_t

using component_t = float

using distance_t = float

Public Functions

explicit IndexHNSW(int d = 0, int M = 32, MetricType metric = METRIC_L2)

explicit IndexHNSW(Index *storage, int M = 32)

~IndexHNSW() override

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void train(idx_t n, const float *x) override

Trains the storage if needed.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

entry point for search

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void reset() override

removes all elements from the database.

void shrink_level_0_neighbors(int size)

void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const

Perform search only on level 0, given the starting points for each vertex.

Parameters:

search_type – 1:perform one search per nprobe, 2: enqueue all entry points

void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)

alternative graph building

void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)

alternative graph building

void reorder_links()

void link_singletons()

void permute_entries(const idx_t *perm)

virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

HNSW hnsw

bool own_fields = false

Index *storage = nullptr

bool init_level0 = true

bool keep_max_size_level0 = false

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexHNSWFlat : public faiss::IndexHNSW

#include <IndexHNSW.h>

Flat index topped with with a HNSW structure to access elements more efficiently.

Public Types

typedef HNSW::storage_idx_t storage_idx_t

using component_t = float

using distance_t = float

Public Functions

IndexHNSWFlat()

IndexHNSWFlat(int d, int M, MetricType metric = METRIC_L2)

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void train(idx_t n, const float *x) override

Trains the storage if needed.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

entry point for search

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void reset() override

removes all elements from the database.

void shrink_level_0_neighbors(int size)

void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const

Perform search only on level 0, given the starting points for each vertex.

Parameters:

search_type – 1:perform one search per nprobe, 2: enqueue all entry points

void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)

alternative graph building

void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)

alternative graph building

void reorder_links()

void link_singletons()

void permute_entries(const idx_t *perm)

virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

HNSW hnsw

bool own_fields = false

Index *storage = nullptr

bool init_level0 = true

bool keep_max_size_level0 = false

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexHNSWPQ : public faiss::IndexHNSW

#include <IndexHNSW.h>

PQ index topped with with a HNSW structure to access elements more efficiently.

Public Types

typedef HNSW::storage_idx_t storage_idx_t

using component_t = float

using distance_t = float

Public Functions

IndexHNSWPQ()

IndexHNSWPQ(int d, int pq_m, int M, int pq_nbits = 8, MetricType metric = METRIC_L2)

virtual void train(idx_t n, const float *x) override

Trains the storage if needed.

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

entry point for search

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void reset() override

removes all elements from the database.

void shrink_level_0_neighbors(int size)

void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const

Perform search only on level 0, given the starting points for each vertex.

Parameters:

search_type – 1:perform one search per nprobe, 2: enqueue all entry points

void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)

alternative graph building

void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)

alternative graph building

void reorder_links()

void link_singletons()

void permute_entries(const idx_t *perm)

virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

HNSW hnsw

bool own_fields = false

Index *storage = nullptr

bool init_level0 = true

bool keep_max_size_level0 = false

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexHNSWSQ : public faiss::IndexHNSW

#include <IndexHNSW.h>

SQ index topped with with a HNSW structure to access elements more efficiently.

Public Types

typedef HNSW::storage_idx_t storage_idx_t

using component_t = float

using distance_t = float

Public Functions

IndexHNSWSQ()

IndexHNSWSQ(int d, ScalarQuantizer::QuantizerType qtype, int M, MetricType metric = METRIC_L2)

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void train(idx_t n, const float *x) override

Trains the storage if needed.

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

entry point for search

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void reset() override

removes all elements from the database.

void shrink_level_0_neighbors(int size)

void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const

Perform search only on level 0, given the starting points for each vertex.

Parameters:

search_type – 1:perform one search per nprobe, 2: enqueue all entry points

void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)

alternative graph building

void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)

alternative graph building

void reorder_links()

void link_singletons()

void permute_entries(const idx_t *perm)

virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

HNSW hnsw

bool own_fields = false

Index *storage = nullptr

bool init_level0 = true

bool keep_max_size_level0 = false

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexHNSW2Level : public faiss::IndexHNSW

#include <IndexHNSW.h>

2-level code structure with fast random access

Public Types

typedef HNSW::storage_idx_t storage_idx_t

using component_t = float

using distance_t = float

Public Functions

IndexHNSW2Level()

IndexHNSW2Level(Index *quantizer, size_t nlist, int m_pq, int M)

void flip_to_ivf()

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

entry point for search

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void train(idx_t n, const float *x) override

Trains the storage if needed.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void reset() override

removes all elements from the database.

void shrink_level_0_neighbors(int size)

void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const

Perform search only on level 0, given the starting points for each vertex.

Parameters:

search_type – 1:perform one search per nprobe, 2: enqueue all entry points

void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)

alternative graph building

void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)

alternative graph building

void reorder_links()

void link_singletons()

void permute_entries(const idx_t *perm)

virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

HNSW hnsw

bool own_fields = false

Index *storage = nullptr

bool init_level0 = true

bool keep_max_size_level0 = false

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

struct IndexHNSWCagra : public faiss::IndexHNSW

Public Types

typedef HNSW::storage_idx_t storage_idx_t

using component_t = float

using distance_t = float

Public Functions

IndexHNSWCagra()

IndexHNSWCagra(int d, int M, MetricType metric = METRIC_L2)

virtual void add(idx_t n, const float *x) override

Add n vectors of dimension d to the index.

Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.

Parameters:

• n – number of vectors

• x – input matrix, size n * d

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

entry point for search

virtual void train(idx_t n, const float *x) override

Trains the storage if needed.

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

Reconstruct a stored vector (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• key – id of the vector to reconstruct

• recons – reconstucted vector (size d)

virtual void reset() override

removes all elements from the database.

void shrink_level_0_neighbors(int size)

void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const

Perform search only on level 0, given the starting points for each vertex.

Parameters:

search_type – 1:perform one search per nprobe, 2: enqueue all entry points

void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)

alternative graph building

void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)

alternative graph building

void reorder_links()

void link_singletons()

void permute_entries(const idx_t *perm)

virtual DistanceComputer *get_distance_computer() const override

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)

Same as add, but stores xids instead of sequential ids.

The default implementation fails with an assertion, as it is not supported by all indexes.

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• xids – if non-null, ids to store for the vectors (size n)

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual size_t remove_ids(const IDSelector &sel)

removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const

Reconstruct vectors i0 to i0 + ni - 1

this function may not be defined for some indexes

Parameters:

• i0 – index of the first vector in the sequence

• ni – number of vectors in the sequence

• recons – reconstucted vector (size ni * d)

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

If there are not enough results for a query, the resulting arrays is padded with -1s.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• k – number of extracted vectors

• distances – output pairwise distances, size n*k

• labels – output labels of the NNs, size n*k

• recons – reconstructed vectors size (n, k, d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual size_t sa_code_size() const

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const

encode a set of vectors

Parameters:

• n – number of vectors

• x – input vectors, size n * d

• bytes – output encoded vectors, size n * sa_code_size()

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void merge_from(Index &otherIndex, idx_t add_id = 0)

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual void check_compatible_for_merge(const Index &otherIndex) const

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

Public Members

bool base_level_only = false

When set to true, the index is immutable. This option is used to copy the knn graph from GpuIndexCagra to the base level of IndexHNSWCagra without adding upper levels. Doing so enables to search the HNSW index, but removes the ability to add vectors.

int num_base_level_search_entrypoints = 32

When base_level_only is set to True, the search function searches only the base level knn graph of the HNSW index. This parameter selects the entry point by randomly selecting some points and using the best one.

HNSW hnsw

bool own_fields = false

Index *storage = nullptr

bool init_level0 = true

bool keep_max_size_level0 = false

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

template<typename IndexT>
struct IndexIDMapTemplate : public IndexT

#include <IndexIDMap.h>

Index that translates search results to ids

Subclassed by faiss::IndexIDMap2Template< IndexT >

Public Types

using component_t = typename IndexT::component_t

using distance_t = typename IndexT::distance_t

Public Functions

explicit IndexIDMapTemplate(IndexT *index)

void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override

Parameters:

xids – if non-null, ids to store for the vectors (size n)

void add(idx_t n, const component_t *x) override

this will fail. Use add_with_ids

void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

void train(idx_t n, const component_t *x) override

void reset() override

size_t remove_ids(const IDSelector &sel) override

remove ids adapted to IndexFlat

void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

void merge_from(IndexT &otherIndex, idx_t add_id = 0) override

void check_compatible_for_merge(const IndexT &otherIndex) const override

size_t sa_code_size() const override

void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

~IndexIDMapTemplate() override

inline IndexIDMapTemplate()

Public Members

IndexT *index = nullptr

bool own_fields = false

! the sub-index

std::vector<idx_t> id_map

! whether pointers are deleted in destructo

template<typename IndexT>
struct IndexIDMap2Template : public faiss::IndexIDMapTemplate<IndexT>

#include <IndexIDMap.h>

same as IndexIDMap but also provides an efficient reconstruction implementation via a 2-way index

Public Types

using component_t = typename IndexT::component_t

using distance_t = typename IndexT::distance_t

Public Functions

explicit IndexIDMap2Template(IndexT *index)

void construct_rev_map()

make the rev_map from scratch

void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override

size_t remove_ids(const IDSelector &sel) override

void reconstruct(idx_t key, component_t *recons) const override

void check_consistency() const

check that the rev_map and the id_map are in sync

void merge_from(IndexT &otherIndex, idx_t add_id = 0) override

inline ~IndexIDMap2Template() override

inline IndexIDMap2Template()

void add(idx_t n, const component_t *x) override

this will fail. Use add_with_ids

void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

void train(idx_t n, const component_t *x) override

void reset() override

void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

void check_compatible_for_merge(const IndexT &otherIndex) const override

size_t sa_code_size() const override

void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override

Public Members

std::unordered_map<idx_t, idx_t> rev_map

IndexT *index = nullptr

bool own_fields = false

! the sub-index

std::vector<idx_t> id_map

! whether pointers are deleted in destructo

struct IDSelectorTranslated : public faiss::IDSelector

Public Functions

inline IDSelectorTranslated(const std::vector<int64_t> &id_map, const IDSelector *sel)

inline IDSelectorTranslated(IndexBinaryIDMap &index_idmap, const IDSelector *sel)

inline IDSelectorTranslated(IndexIDMap &index_idmap, const IDSelector *sel)

inline virtual bool is_member(idx_t id) const override

Public Members

const std::vector<int64_t> &id_map

const IDSelector *sel

struct Level1Quantizer

#include <IndexIVF.h>

Encapsulates a quantizer object for the IndexIVF

The class isolates the fields that are independent of the storage of the lists (especially training)

Subclassed by faiss::IndexIVFInterface, faiss::IndexShardsIVF

Public Functions

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)

Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const

compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const

idx_t decode_listno(const uint8_t *code) const

Level1Quantizer(Index *quantizer, size_t nlist)

Level1Quantizer()

~Level1Quantizer()

Public Members

Index *quantizer = nullptr

quantizer that maps vectors to inverted lists

size_t nlist = 0

number of inverted lists

char quantizer_trains_alone = 0

= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer

bool own_fields = false

whether object owns the quantizer

ClusteringParameters cp

to override default clustering params

Index *clustering_index = nullptr

to override index used during clustering

struct SearchParametersIVF : public faiss::SearchParameters

Subclassed by faiss::IVFPQSearchParameters

Public Functions

inline virtual ~SearchParametersIVF()

Public Members

size_t nprobe = 1

number of probes at query time

size_t max_codes = 0

max nb of codes to visit to do a query

SearchParameters *quantizer_params = nullptr

void *inverted_list_context = nullptr

context object to pass to InvertedLists

IDSelector *sel = nullptr

if non-null, only these IDs will be considered during search.

struct IndexIVFInterface : public faiss::Level1Quantizer

Subclassed by faiss::IndexIVF, faiss::gpu::GpuIndexIVF

Public Functions

inline explicit IndexIVFInterface(Index *quantizer = nullptr, size_t nlist = 0)

virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const = 0

search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• distance – output distances, size n * k

• labels – output labels, size n * k

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const = 0

Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• result – Output results

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

inline virtual ~IndexIVFInterface()

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)

Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const

compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const

idx_t decode_listno(const uint8_t *code) const

Public Members

size_t nprobe = 1

number of probes at query time

size_t max_codes = 0

max nb of codes to visit to do a query

Index *quantizer = nullptr

quantizer that maps vectors to inverted lists

size_t nlist = 0

number of inverted lists

char quantizer_trains_alone = 0

= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer

bool own_fields = false

whether object owns the quantizer

ClusteringParameters cp

to override default clustering params

Index *clustering_index = nullptr

to override index used during clustering

struct IndexIVF : public faiss::Index, public faiss::IndexIVFInterface

#include <IndexIVF.h>

Index based on a inverted file (IVF)

In the inverted file, the quantizer (an Index instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.

The inverted list object is required only after trainng. If none is set externally, an ArrayInvertedLists is used automatically.

At search time, the vector to be searched is also quantized, and only the list corresponding to the quantization index is searched. This speeds up the search by making it non-exhaustive. This can be relaxed using multi-probe search: a few (nprobe) quantization indices are selected and several inverted lists are visited.

Sub-classes implement a post-filtering of the index that refines the distance estimation from the query to databse vectors.

Subclassed by faiss::IndexIVFAdditiveQuantizer, faiss::IndexIVFFastScan, faiss::IndexIVFFlat, faiss::IndexIVFPQ, faiss::IndexIVFScalarQuantizer, faiss::IndexIVFSpectralHash

Public Types

using component_t = float

using distance_t = float

Public Functions

IndexIVF(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)

The Inverted file takes a quantizer (an Index) on input, which implements the function mapping a vector to a list identifier.

virtual void reset() override

removes all elements from the database.

virtual void train(idx_t n, const float *x) override

Trains the quantizer and calls train_encoder to train sub-quantizers.

virtual void add(idx_t n, const float *x) override

Calls add_with_ids with NULL ids.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override

default implementation that calls encode_vectors

virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)

Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.

Parameters:

precomputed_idx – quantization indices for the input vectors (size n)

virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const = 0

Encodes a set of vectors as they would appear in the inverted lists

Parameters:

• list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.

• codes – output codes, size n * code_size

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

virtual void train_encoder(idx_t n, const float *x, const idx_t *assign)

Train the encoder for the vectors.

If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr

virtual idx_t train_encoder_num_vectors() const

can be redefined by subclasses to indicate how many training vectors they need

virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• distance – output distances, size n * k

• labels – output labels, size n * k

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• result – Output results

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

assign the vectors, then call search_preassign

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const

Get a scanner for this index (store_pairs means ignore labels)

The default search implementation uses this to compute the distances

virtual void reconstruct(idx_t key, float *recons) const override

reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2

virtual void update_vectors(int nv, const idx_t *idx, const float *v)

Update a subset of vectors.

The index must have a direct_map

Parameters:

• nv – nb of vectors to update

• idx – vector indices to update, size nv

• v – vectors of new values, size nv*d

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct a subset of the indexed vectors.

Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.

Parameters:

• i0 – first vector to reconstruct

• ni – nb of vectors to reconstruct

• recons – output array of reconstructed vectors, size ni * d

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the store_pairs flag in search_preassigned().

Parameters:

recons – reconstructed vectors size (n, k, d)

void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const

Similar to search, but also returns the codes corresponding to the stored vectors for the search results.

Parameters:

• codes – codes (n, k, code_size)

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const

Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.

Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with store_pairs set.

virtual size_t remove_ids(const IDSelector &sel) override

Dataset manipulation functions.

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual CodePacker *get_CodePacker() const

virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const

copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type

~IndexIVF() override

inline size_t get_list_size(size_t list_no) const

bool check_ids_sorted() const

are the ids sorted?

void make_direct_map(bool new_maintain_direct_map = true)

initialize a direct map

Parameters:

new_maintain_direct_map – if true, create a direct map, else clear it

void set_direct_map_type(DirectMap::Type type)

void replace_invlists(InvertedLists *il, bool own = false)

replace the inverted lists, old one is deallocated if own_invlists

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors sa_encode will call encode_vector with include_listno=true

Parameters:

• n – nb of vectors to encode

• x – the vectors to encode

• bytes – output array for the codes

Returns:

nb of bytes written to codes

IndexIVF()

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)

Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const

compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const

idx_t decode_listno(const uint8_t *code) const

Public Members

InvertedLists *invlists = nullptr

Access to the actual data.

bool own_invlists = false

size_t code_size = 0

code size per vector in bytes

int parallel_mode = 0

Parallel mode determines how queries are parallelized with OpenMP

0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity

PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized

const int PARALLEL_MODE_NO_HEAP_INIT = 1024

DirectMap direct_map

optional map that maps back ids to invlist entries. This enables reconstruct()

bool by_residual = true

do the codes in the invlists encode the vectors relative to the centroids?

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

size_t nprobe = 1

number of probes at query time

size_t max_codes = 0

max nb of codes to visit to do a query

Index *quantizer = nullptr

quantizer that maps vectors to inverted lists

size_t nlist = 0

number of inverted lists

char quantizer_trains_alone = 0

= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer

bool own_fields = false

whether object owns the quantizer

ClusteringParameters cp

to override default clustering params

Index *clustering_index = nullptr

to override index used during clustering

struct InvertedListScanner

#include <IndexIVF.h>

Object that handles a query. The inverted lists to scan are provided externally. The object has a lot of state, but distance_to_code and scan_codes can be called in multiple threads

Public Functions

inline InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr)

virtual void set_query(const float *query_vector) = 0

from now on we handle this query.

virtual void set_list(idx_t list_no, float coarse_dis) = 0

following codes come from this inverted list

virtual float distance_to_code(const uint8_t *code) const = 0

compute a single query-to-code distance

virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, float *distances, idx_t *labels, size_t k) const

scan a set of codes, compute distances to current query and update heap of results if necessary. Default implementation calls distance_to_code.

Parameters:

• n – number of codes to scan

• codes – codes to scan (n * code_size)

• ids – corresponding ids (ignored if store_pairs)

• distances – heap distances (size k)

• labels – heap labels (size k)

• k – heap size

Returns:

number of heap updates performed

virtual size_t iterate_codes(InvertedListsIterator *iterator, float *distances, idx_t *labels, size_t k, size_t &list_size) const

virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, float radius, RangeQueryResult &result) const

scan a set of codes, compute distances to current query and update results if distances are below radius

(default implementation fails)

virtual void iterate_codes_range(InvertedListsIterator *iterator, float radius, RangeQueryResult &result, size_t &list_size) const

inline virtual ~InvertedListScanner()

Public Members

idx_t list_no = -1

remember current list

bool keep_max = false

keep maximum instead of minimum

bool store_pairs

store positions in invlists rather than labels

const IDSelector *sel

search in this subset of ids

size_t code_size = 0

used in default implementation of scan_codes

struct IndexIVFStats

Public Functions

inline IndexIVFStats()

void reset()

void add(const IndexIVFStats &other)

Public Members

size_t nq

size_t nlist

size_t ndis

size_t nheap_updates

double quantization_time

double search_time

struct IndexIVFAdditiveQuantizer : public faiss::IndexIVF

#include <IndexIVFAdditiveQuantizer.h>

Abstract class for IVF additive quantizers. The search functions are in common.

Subclassed by faiss::IndexIVFLocalSearchQuantizer, faiss::IndexIVFProductLocalSearchQuantizer, faiss::IndexIVFProductResidualQuantizer, faiss::IndexIVFResidualQuantizer

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq, Index *quantizer, size_t d, size_t nlist, MetricType metric = METRIC_L2)

explicit IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq)

virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override

Train the encoder for the vectors.

If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr

virtual idx_t train_encoder_num_vectors() const override

can be redefined by subclasses to indicate how many training vectors they need

virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override

Encodes a set of vectors as they would appear in the inverted lists

Parameters:

• list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.

• codes – output codes, size n * code_size

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override

Get a scanner for this index (store_pairs means ignore labels)

The default search implementation uses this to compute the distances

virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override

Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.

Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with store_pairs set.

~IndexIVFAdditiveQuantizer() override

virtual void reset() override

removes all elements from the database.

virtual void train(idx_t n, const float *x) override

Trains the quantizer and calls train_encoder to train sub-quantizers.

virtual void add(idx_t n, const float *x) override

Calls add_with_ids with NULL ids.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override

default implementation that calls encode_vectors

virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)

Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.

Parameters:

precomputed_idx – quantization indices for the input vectors (size n)

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• distance – output distances, size n * k

• labels – output labels, size n * k

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• result – Output results

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

assign the vectors, then call search_preassign

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2

virtual void update_vectors(int nv, const idx_t *idx, const float *v)

Update a subset of vectors.

The index must have a direct_map

Parameters:

• nv – nb of vectors to update

• idx – vector indices to update, size nv

• v – vectors of new values, size nv*d

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct a subset of the indexed vectors.

Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.

Parameters:

• i0 – first vector to reconstruct

• ni – nb of vectors to reconstruct

• recons – output array of reconstructed vectors, size ni * d

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the store_pairs flag in search_preassigned().

Parameters:

recons – reconstructed vectors size (n, k, d)

void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const

Similar to search, but also returns the codes corresponding to the stored vectors for the search results.

Parameters:

• codes – codes (n, k, code_size)

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual size_t remove_ids(const IDSelector &sel) override

Dataset manipulation functions.

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual CodePacker *get_CodePacker() const

virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const

copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type

inline size_t get_list_size(size_t list_no) const

bool check_ids_sorted() const

are the ids sorted?

void make_direct_map(bool new_maintain_direct_map = true)

initialize a direct map

Parameters:

new_maintain_direct_map – if true, create a direct map, else clear it

void set_direct_map_type(DirectMap::Type type)

void replace_invlists(InvertedLists *il, bool own = false)

replace the inverted lists, old one is deallocated if own_invlists

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors sa_encode will call encode_vector with include_listno=true

Parameters:

• n – nb of vectors to encode

• x – the vectors to encode

• bytes – output array for the codes

Returns:

nb of bytes written to codes

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)

Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const

compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const

idx_t decode_listno(const uint8_t *code) const

Public Members

AdditiveQuantizer *aq

int use_precomputed_table = 0

InvertedLists *invlists = nullptr

Access to the actual data.

bool own_invlists = false

size_t code_size = 0

code size per vector in bytes

int parallel_mode = 0

Parallel mode determines how queries are parallelized with OpenMP

0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity

PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized

const int PARALLEL_MODE_NO_HEAP_INIT = 1024

DirectMap direct_map

optional map that maps back ids to invlist entries. This enables reconstruct()

bool by_residual = true

do the codes in the invlists encode the vectors relative to the centroids?

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

size_t nprobe = 1

number of probes at query time

size_t max_codes = 0

max nb of codes to visit to do a query

Index *quantizer = nullptr

quantizer that maps vectors to inverted lists

size_t nlist = 0

number of inverted lists

char quantizer_trains_alone = 0

= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer

bool own_fields = false

whether object owns the quantizer

ClusteringParameters cp

to override default clustering params

Index *clustering_index = nullptr

to override index used during clustering

struct IndexIVFResidualQuantizer : public faiss::IndexIVFAdditiveQuantizer

#include <IndexIVFAdditiveQuantizer.h>

IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

IndexIVFResidualQuantizer()

virtual ~IndexIVFResidualQuantizer()

virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override

Train the encoder for the vectors.

If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr

virtual idx_t train_encoder_num_vectors() const override

can be redefined by subclasses to indicate how many training vectors they need

virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override

Encodes a set of vectors as they would appear in the inverted lists

Parameters:

• list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.

• codes – output codes, size n * code_size

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override

Get a scanner for this index (store_pairs means ignore labels)

The default search implementation uses this to compute the distances

virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override

Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.

Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with store_pairs set.

virtual void reset() override

removes all elements from the database.

virtual void train(idx_t n, const float *x) override

Trains the quantizer and calls train_encoder to train sub-quantizers.

virtual void add(idx_t n, const float *x) override

Calls add_with_ids with NULL ids.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override

default implementation that calls encode_vectors

virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)

Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.

Parameters:

precomputed_idx – quantization indices for the input vectors (size n)

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• distance – output distances, size n * k

• labels – output labels, size n * k

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• result – Output results

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

assign the vectors, then call search_preassign

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2

virtual void update_vectors(int nv, const idx_t *idx, const float *v)

Update a subset of vectors.

The index must have a direct_map

Parameters:

• nv – nb of vectors to update

• idx – vector indices to update, size nv

• v – vectors of new values, size nv*d

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct a subset of the indexed vectors.

Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.

Parameters:

• i0 – first vector to reconstruct

• ni – nb of vectors to reconstruct

• recons – output array of reconstructed vectors, size ni * d

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the store_pairs flag in search_preassigned().

Parameters:

recons – reconstructed vectors size (n, k, d)

void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const

Similar to search, but also returns the codes corresponding to the stored vectors for the search results.

Parameters:

• codes – codes (n, k, code_size)

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual size_t remove_ids(const IDSelector &sel) override

Dataset manipulation functions.

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual CodePacker *get_CodePacker() const

virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const

copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type

inline size_t get_list_size(size_t list_no) const

bool check_ids_sorted() const

are the ids sorted?

void make_direct_map(bool new_maintain_direct_map = true)

initialize a direct map

Parameters:

new_maintain_direct_map – if true, create a direct map, else clear it

void set_direct_map_type(DirectMap::Type type)

void replace_invlists(InvertedLists *il, bool own = false)

replace the inverted lists, old one is deallocated if own_invlists

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors sa_encode will call encode_vector with include_listno=true

Parameters:

• n – nb of vectors to encode

• x – the vectors to encode

• bytes – output array for the codes

Returns:

nb of bytes written to codes

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)

Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const

compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const

idx_t decode_listno(const uint8_t *code) const

Public Members

ResidualQuantizer rq

The residual quantizer used to encode the vectors.

AdditiveQuantizer *aq

int use_precomputed_table = 0

InvertedLists *invlists = nullptr

Access to the actual data.

bool own_invlists = false

size_t code_size = 0

code size per vector in bytes

int parallel_mode = 0

Parallel mode determines how queries are parallelized with OpenMP

0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity

PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized

const int PARALLEL_MODE_NO_HEAP_INIT = 1024

DirectMap direct_map

optional map that maps back ids to invlist entries. This enables reconstruct()

bool by_residual = true

do the codes in the invlists encode the vectors relative to the centroids?

int d

vector dimension

idx_t ntotal

total nb of indexed vectors

bool verbose

verbosity level

bool is_trained

set if the Index does not require training, or if training is done already

MetricType metric_type

type of metric this index uses for search

float metric_arg

argument of the metric type

size_t nprobe = 1

number of probes at query time

size_t max_codes = 0

max nb of codes to visit to do a query

Index *quantizer = nullptr

quantizer that maps vectors to inverted lists

size_t nlist = 0

number of inverted lists

char quantizer_trains_alone = 0

= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer

bool own_fields = false

whether object owns the quantizer

ClusteringParameters cp

to override default clustering params

Index *clustering_index = nullptr

to override index used during clustering

struct IndexIVFLocalSearchQuantizer : public faiss::IndexIVFAdditiveQuantizer

#include <IndexIVFAdditiveQuantizer.h>

IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.

Public Types

using Search_type_t = AdditiveQuantizer::Search_type_t

using component_t = float

using distance_t = float

Public Functions

IndexIVFLocalSearchQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)

Constructor.

Parameters:

• d – dimensionality of the input vectors

• M – number of subquantizers

• nbits – number of bit per subvector index

IndexIVFLocalSearchQuantizer()

virtual ~IndexIVFLocalSearchQuantizer()

virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override

Train the encoder for the vectors.

If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr

virtual idx_t train_encoder_num_vectors() const override

can be redefined by subclasses to indicate how many training vectors they need

virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override

Encodes a set of vectors as they would appear in the inverted lists

Parameters:

• list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.

• codes – output codes, size n * code_size

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override

Get a scanner for this index (store_pairs means ignore labels)

The default search implementation uses this to compute the distances

virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override

decode a set of vectors

Parameters:

• n – number of vectors

• bytes – input encoded vectors, size n * sa_code_size()

• x – output vectors, size n * d

virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override

Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.

Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with store_pairs set.

virtual void reset() override

removes all elements from the database.

virtual void train(idx_t n, const float *x) override

Trains the quantizer and calls train_encoder to train sub-quantizers.

virtual void add(idx_t n, const float *x) override

Calls add_with_ids with NULL ids.

virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override

default implementation that calls encode_vectors

virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)

Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.

Parameters:

precomputed_idx – quantization indices for the input vectors (size n)

virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override

Add vectors that are computed with the standalone codec

Parameters:

• codes – codes to add size n * sa_code_size()

• xids – corresponding ids, size n

virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• distance – output distances, size n * k

• labels – output labels, size n * k

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override

Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.

Parameters:

• n – nb of vectors to query

• x – query vectors, size nx * d

• assign – coarse quantization indices, size nx * nprobe

• centroid_dis – distances to coarse centroids, size nx * nprobe

• result – Output results

• store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).

• params – used to override the object’s search parameters

• stats – search stats to be updated (can be null)

virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

assign the vectors, then call search_preassign

virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override

query n vectors of dimension d to the index.

return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• radius – search radius

• result – result table

virtual void reconstruct(idx_t key, float *recons) const override

reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2

virtual void update_vectors(int nv, const idx_t *idx, const float *v)

Update a subset of vectors.

The index must have a direct_map

Parameters:

• nv – nb of vectors to update

• idx – vector indices to update, size nv

• v – vectors of new values, size nv*d

virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override

Reconstruct a subset of the indexed vectors.

Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.

Parameters:

• i0 – first vector to reconstruct

• ni – nb of vectors to reconstruct

• recons – output array of reconstructed vectors, size ni * d

virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override

Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.

Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the store_pairs flag in search_preassigned().

Parameters:

recons – reconstructed vectors size (n, k, d)

void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const

Similar to search, but also returns the codes corresponding to the stored vectors for the search results.

Parameters:

• codes – codes (n, k, code_size)

• include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)

virtual size_t remove_ids(const IDSelector &sel) override

Dataset manipulation functions.

virtual void check_compatible_for_merge(const Index &otherIndex) const override

check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.

virtual void merge_from(Index &otherIndex, idx_t add_id) override

moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)

virtual CodePacker *get_CodePacker() const

virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const

copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type

inline size_t get_list_size(size_t list_no) const

bool check_ids_sorted() const

are the ids sorted?

void make_direct_map(bool new_maintain_direct_map = true)

initialize a direct map

Parameters:

new_maintain_direct_map – if true, create a direct map, else clear it

void set_direct_map_type(DirectMap::Type type)

void replace_invlists(InvertedLists *il, bool own = false)

replace the inverted lists, old one is deallocated if own_invlists

virtual size_t sa_code_size() const override

size of the produced codes in bytes

virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override

encode a set of vectors sa_encode will call encode_vector with include_listno=true

Parameters:

• n – nb of vectors to encode

• x – the vectors to encode

• bytes – output array for the codes

Returns:

nb of bytes written to codes

virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const

return the indexes of the k vectors closest to the query x.

This function is identical as search but only return labels of neighbors.

Parameters:

• n – number of vectors

• x – input vectors to search, size n * d

• labels – output labels of the NNs, size n*k

• k – number of nearest neighbours

virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const

Reconstruct several stored vectors (or an approximation if lossy coding)

this function may not be defined for some indexes

Parameters:

• n – number of vectors to reconstruct

• keys – ids of the vectors to reconstruct (size n)

• recons – reconstucted vector (size n * d)

virtual void compute_residual(const float *x, float *residual, idx_t key) const

Computes a residual vector after indexing encoding.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• x – input vector, size d

• residual – output residual vector, size d

• key – encoded index, as returned by search and assign

virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const

Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.

The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.

Parameters:

• n – number of vectors

• xs – input vectors, size (n x d)

• residuals – output residual vectors, size (n x d)

• keys – encoded index, as returned by search and assign

virtual DistanceComputer *get_distance_computer() const

Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.

DistanceComputer is implemented for indexes that support random access of their vectors.

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)

Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const

compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const

idx_t decode_listno(const uint8_t *code) const

Public Members

LocalSearchQuantizer lsq

The LSQ quantizer used to encode the vectors.

AdditiveQuantizer *aq

int use_precomputed_table =