Namespace faiss
-
namespace faiss
Implementation of k-means clustering with many variants.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.
IDSelector is intended to define a subset of vectors to handle (for removal or as subset to search)
PQ4 SIMD packing and accumulation functions
The basic kernel accumulates nq query vectors with bbs = nb * 2 * 16 vectors and produces an output matrix for that. It is interesting for nq * nb <= 4, otherwise register spilling becomes too large.
The implementation of these functions is spread over 3 cpp files to reduce parallel compile times. Templates are instantiated explicitly.
This file contains callbacks for kernels that compute distances.
Throughout the library, vectors are provided as float * pointers. Most algorithms can be optimized when several vectors are processed (added/searched) together in a batch. In this case, they are passed in as a matrix. When n vectors of size d are provided as float * x, component j of vector i is
x[ i * d + j ]
where 0 <= i < n and 0 <= j < d. In other words, matrices are always compact. When specifying the size of the matrix, we call it an n*d matrix, which implies a row-major storage.
I/O functions can read/write to a filename, a file handle or to an object that abstracts the medium.
The read functions return objects that should be deallocated with delete. All references within these objectes are owned by the object.
Definition of inverted lists + a few common classes that implement the interface.
Since IVF (inverted file) indexes are of so much use for large-scale use cases, we group a few functions related to them in this small library. Most functions work both on IndexIVFs and IndexIVFs embedded within an IndexPreTransform.
In this file are the implementations of extra metrics beyond L2 and inner product
Implements a few neural net layers, mainly to support QINCo
Defines a few objects that apply transformations to a set of vectors Often these are pre-processing steps.
Typedefs
-
using IndexIDMap = IndexIDMapTemplate<Index>
-
using IndexBinaryIDMap = IndexIDMapTemplate<IndexBinary>
-
using IndexIDMap2 = IndexIDMap2Template<Index>
-
using IndexBinaryIDMap2 = IndexIDMap2Template<IndexBinary>
-
using IVFSearchParameters = SearchParametersIVF
-
using IndexReplicas = IndexReplicasTemplate<Index>
-
using IndexBinaryReplicas = IndexReplicasTemplate<IndexBinary>
-
using IndexShards = IndexShardsTemplate<Index>
-
using IndexBinaryShards = IndexShardsTemplate<IndexBinary>
-
using ConcatenatedInvertedLists = HStackInvertedLists
-
using idx_t = int64_t
all vector indices are this type
Enums
-
enum MetricType
The metric space for vector comparison for Faiss indices and algorithms.
Most algorithms support both inner product and L2, with the flat (brute-force) indices supporting additional metric types for vector comparison.
Values:
-
enumerator METRIC_INNER_PRODUCT
maximum inner product search
-
enumerator METRIC_L2
squared L2 search
-
enumerator METRIC_L1
L1 (aka cityblock)
-
enumerator METRIC_Linf
infinity distance
-
enumerator METRIC_Lp
L_p distance, p is given by a faiss::Index metric_arg
-
enumerator METRIC_Canberra
some additional metrics defined in scipy.spatial.distance
-
enumerator METRIC_BrayCurtis
-
enumerator METRIC_JensenShannon
-
enumerator METRIC_Jaccard
sum_i(min(a_i, b_i)) / sum_i(max(a_i, b_i)) where a_i, b_i > 0
-
enumerator METRIC_NaNEuclidean
Squared Eucliden distance, ignoring NaNs.
-
enumerator METRIC_ABS_INNER_PRODUCT
abs(x | y): the distance to a hyperplane
-
enumerator METRIC_INNER_PRODUCT
Functions
-
IndexBinary *clone_binary_index(const IndexBinary *index)
-
float kmeans_clustering(size_t d, size_t n, size_t k, const float *x, float *centroids)
simplified interface
- Parameters:
d – dimension of the data
n – nb of training vectors
k – nb of output centroids
x – training set (size n * d)
centroids – output centroids (size k * d)
- Returns:
final quantization error
-
std::string reverse_index_factory(const faiss::IndexBinary *index)
-
template<typename PQDecoderT>
inline float distance_single_code_generic(const size_t M, const size_t nbits, const float *sim_table, const uint8_t *code) Returns the distance to a single code.
- template<typename PQDecoderT> inline void distance_four_codes_generic (const size_t M, const size_t nbits, const float *sim_table, const uint8_t *__restrict code0, const uint8_t *__restrict code1, const uint8_t *__restrict code2, const uint8_t *__restrict code3, float &result0, float &result1, float &result2, float &result3)
Combines 4 operations of distance_single_code() General-purpose version.
-
template<typename PQDecoderT>
inline float distance_single_code(const size_t M, const size_t nbits, const float *sim_table, const uint8_t *code)
- template<typename PQDecoderT> inline void distance_four_codes (const size_t M, const size_t nbits, const float *sim_table, const uint8_t *__restrict code0, const uint8_t *__restrict code1, const uint8_t *__restrict code2, const uint8_t *__restrict code3, float &result0, float &result1, float &result2, float &result3)
-
void handleExceptions(std::vector<std::pair<int, std::exception_ptr>> &exceptions)
Handle multiple exceptions from worker threads, throwing an appropriate exception that aggregates the information The pair int is the thread that generated the exception
-
std::string demangle_cpp_symbol(const char *name)
make typeids more readable
-
int search_from_candidates(const HNSW &hnsw, DistanceComputer &qdis, ResultHandler<HNSW::C> &res, HNSW::MinimaxHeap &candidates, VisitedTable &vt, HNSWStats &stats, int level, int nres_in = 0, const SearchParametersHNSW *params = nullptr)
-
HNSWStats greedy_update_nearest(const HNSW &hnsw, DistanceComputer &qdis, int level, HNSW::storage_idx_t &nearest, float &d_nearest)
-
std::priority_queue<HNSW::Node> search_from_candidate_unbounded(const HNSW &hnsw, const HNSW::Node &node, DistanceComputer &qdis, int ef, VisitedTable *vt, HNSWStats &stats)
-
void search_neighbors_to_add(HNSW &hnsw, DistanceComputer &qdis, std::priority_queue<HNSW::NodeDistCloser> &results, int entry_point, float d_entry_point, int level, VisitedTable &vt, bool reference_version = false)
-
ArrayInvertedLists *set_array_invlist(IndexIVF *ivf, std::vector<std::vector<idx_t>> &ids)
-
void read_ProductQuantizer(ProductQuantizer *pq, IOReader *f)
-
void read_ScalarQuantizer(ScalarQuantizer *ivsc, IOReader *f)
-
uint32_t fourcc(const char sx[4])
cast a 4-character string to a uint32_t that can be written and read easily
-
uint32_t fourcc(const std::string &sx)
-
void fourcc_inv(uint32_t x, char str[5])
-
std::string fourcc_inv(uint32_t x)
-
std::string fourcc_inv_printable(uint32_t x)
-
void smawk(const idx_t nrows, const idx_t ncols, const float *x, idx_t *argmins)
SMAWK algorithm. Find the row minima of a monotone matrix.
Expose this for testing.
- Parameters:
nrows – number of rows
ncols – number of columns
x – input matrix, size (nrows, ncols)
argmins – argmin of each row
-
double kmeans1d(const float *x, size_t n, size_t nclusters, float *centroids)
Exact 1D K-Means by dynamic programming
From “Fast Exact k-Means, k-Medians and Bregman Divergence Clustering in 1D” Allan Grønlund, Kasper Green Larsen, Alexander Mathiasen, Jesper Sindahl Nielsen, Stefan Schneider, Mingzhou Song, ArXiV’17
Section 2.2
https://arxiv.org/abs/1701.07204
- Parameters:
x – input 1D array
n – input array length
nclusters – number of clusters
centroids – output centroids, size nclusters
- Returns:
imbalancce factor
-
void pq4_pack_codes(const uint8_t *codes, size_t ntotal, size_t M, size_t nb, size_t bbs, size_t nsq, uint8_t *blocks)
Pack codes for consumption by the SIMD kernels. The unused bytes are set to 0.
- Parameters:
codes – input codes, size (ntotal, ceil(M / 2))
ntotal – number of input codes
nb – output number of codes (ntotal rounded up to a multiple of bbs)
nsq – number of sub-quantizers (=M rounded up to a muliple of 2)
bbs – size of database blocks (multiple of 32)
blocks – output array, size nb * nsq / 2.
-
void pq4_pack_codes_range(const uint8_t *codes, size_t M, size_t i0, size_t i1, size_t bbs, size_t nsq, uint8_t *blocks)
Same as pack_codes but write in a given range of the output, leaving the rest untouched. Assumes allocated entries are 0 on input.
- Parameters:
codes – input codes, size (i1 - i0, ceil(M / 2))
i0 – first output code to write
i1 – last output code to write
blocks – output array, size at least ceil(i1 / bbs) * bbs * nsq / 2
-
uint8_t pq4_get_packed_element(const uint8_t *data, size_t bbs, size_t nsq, size_t vector_id, size_t sq)
get a single element from a packed codes table
- Parameters:
vector_id – vector id
sq – subquantizer (< nsq)
-
void pq4_set_packed_element(uint8_t *data, uint8_t code, size_t bbs, size_t nsq, size_t vector_id, size_t sq)
set a single element “code” into a packed codes table
- Parameters:
vector_id – vector id
sq – subquantizer (< nsq)
-
void pq4_pack_LUT(int nq, int nsq, const uint8_t *src, uint8_t *dest)
Pack Look-up table for consumption by the kernel.
- Parameters:
nq – number of queries
nsq – number of sub-quantizers (muliple of 2)
src – input array, size (nq, 16)
dest – output array, size (nq, 16)
-
void pq4_accumulate_loop(int nq, size_t nb, int bbs, int nsq, const uint8_t *codes, const uint8_t *LUT, SIMDResultHandler &res, const NormTableScaler *scaler)
Loop over database elements and accumulate results into result handler
- Parameters:
nq – number of queries
nb – number of database elements
bbs – size of database blocks (multiple of 32)
nsq – number of sub-quantizers (muliple of 2)
codes – packed codes array
LUT – packed look-up table
scaler – scaler to scale the encoded norm
-
int pq4_qbs_to_nq(int qbs)
-
int pq4_preferred_qbs(int nq)
return the preferred decomposition in blocks for a nb of queries.
-
int pq4_pack_LUT_qbs(int fqbs, int nsq, const uint8_t *src, uint8_t *dest)
Pack Look-up table for consumption by the kernel.
- Parameters:
qbs – 4-bit encoded number of query blocks, the total number of queries handled (nq) is deduced from it
nsq – number of sub-quantizers (muliple of 2)
src – input array, size (nq, 16)
dest – output array, size (nq, 16)
- Returns:
nq
-
int pq4_pack_LUT_qbs_q_map(int qbs, int nsq, const uint8_t *src, const int *q_map, uint8_t *dest)
Same as pq4_pack_LUT_qbs, except the source vectors are remapped with q_map
-
void pq4_accumulate_loop_qbs(int qbs, size_t nb, int nsq, const uint8_t *codes, const uint8_t *LUT, SIMDResultHandler &res, const NormTableScaler *scaler = nullptr)
Run accumulation loop.
- Parameters:
qbs – 4-bit encoded number of queries
nb – number of database codes (mutliple of bbs)
nsq – number of sub-quantizers
codes – encoded database vectors (packed)
LUT – look-up table (packed)
res – call-back for the resutls
scaler – scaler to scale the encoded norm
-
void beam_search_encode_step(size_t d, size_t K, const float *cent, size_t n, size_t beam_size, const float *residuals, size_t m, const int32_t *codes, size_t new_beam_size, int32_t *new_codes, float *new_residuals, float *new_distances, Index *assign_index = nullptr, ApproxTopK_mode_t approx_topk = ApproxTopK_mode_t::EXACT_TOPK)
Encode a residual by sampling from a centroid table.
This is a single encoding step the residual quantizer. It allows low-level access to the encoding function, exposed mainly for unit tests.
- Parameters:
n – number of vectors to handle
residuals – vectors to encode, size (n, beam_size, d)
cent – centroids, size (K, d)
beam_size – input beam size
m – size of the codes for the previous encoding steps
codes – code array for the previous steps of the beam (n, beam_size, m)
new_beam_size – output beam size (should be <= K * beam_size)
new_codes – output codes, size (n, new_beam_size, m + 1)
new_residuals – output residuals, size (n, new_beam_size, d)
new_distances – output distances, size (n, new_beam_size)
assign_index – if non-NULL, will be used to perform assignment
-
void beam_search_encode_step_tab(size_t K, size_t n, size_t beam_size, const float *codebook_cross_norms, size_t ldc, const uint64_t *codebook_offsets, const float *query_cp, size_t ldqc, const float *cent_norms_i, size_t m, const int32_t *codes, const float *distances, size_t new_beam_size, int32_t *new_codes, float *new_distances, ApproxTopK_mode_t approx_topk = ApproxTopK_mode_t::EXACT_TOPK)
Encode a set of vectors using their dot products with the codebooks
- Parameters:
K – number of vectors in the codebook
n – nb of vectors to encode
beam_size – input beam size
codebook_cross_norms – inner product of this codebook with the m previously encoded codebooks
codebook_offsets – offsets into codebook_cross_norms for each previous codebook
query_cp – dot products of query vectors with ???
cent_norms_i – norms of centroids
-
template<class Consumer, class ...Types>
Consumer::T dispatch_knn_ResultHandler(size_t nx, float *vals, int64_t *ids, size_t k, MetricType metric, const IDSelector *sel, Consumer &consumer, Types... args)
-
template<class Consumer, class ...Types>
Consumer::T dispatch_range_ResultHandler(RangeSearchResult *res, float radius, MetricType metric, const IDSelector *sel, Consumer &consumer, Types... args)
-
Index *index_factory(int d, const char *description, MetricType metric = METRIC_L2)
Build and index with the sequence of processing steps described in the string.
-
IndexBinary *index_binary_factory(int d, const char *description)
-
void write_index(const Index *idx, const char *fname, int io_flags = 0)
-
void write_index(const Index *idx, FILE *f, int io_flags = 0)
-
void write_index_binary(const IndexBinary *idx, const char *fname)
-
void write_index_binary(const IndexBinary *idx, FILE *f)
-
void write_index_binary(const IndexBinary *idx, IOWriter *writer)
-
Index *read_index(const char *fname, int io_flags = 0)
-
Index *read_index(FILE *f, int io_flags = 0)
-
IndexBinary *read_index_binary(const char *fname, int io_flags = 0)
-
IndexBinary *read_index_binary(FILE *f, int io_flags = 0)
-
IndexBinary *read_index_binary(IOReader *reader, int io_flags = 0)
-
void write_VectorTransform(const VectorTransform *vt, const char *fname)
-
void write_VectorTransform(const VectorTransform *vt, IOWriter *f)
-
VectorTransform *read_VectorTransform(const char *fname)
-
VectorTransform *read_VectorTransform(IOReader *f)
-
ProductQuantizer *read_ProductQuantizer(const char *fname)
-
ProductQuantizer *read_ProductQuantizer(IOReader *reader)
-
void write_ProductQuantizer(const ProductQuantizer *pq, const char *fname)
-
void write_ProductQuantizer(const ProductQuantizer *pq, IOWriter *f)
-
void write_InvertedLists(const InvertedLists *ils, IOWriter *f)
-
InvertedLists *read_InvertedLists(IOReader *reader, int io_flags = 0)
-
void initialize_IVFPQ_precomputed_table(int &use_precomputed_table, const Index *quantizer, const ProductQuantizer &pq, AlignedTable<float> &precomputed_table, bool by_residual, bool verbose)
Pre-compute distance tables for IVFPQ with by-residual and METRIC_L2
- Parameters:
use_precomputed_table – (I/O) =-1: force disable =0: decide heuristically (default: use tables only if they are < precomputed_tables_max_bytes), set use_precomputed_table on output =1: tables that work for all quantizers (size 256 * nlist * M) =2: specific version for MultiIndexQuantizer (much more compact)
precomputed_table – precomputed table to initialize
-
inline uint64_t lo_build(uint64_t list_id, uint64_t offset)
-
inline uint64_t lo_listno(uint64_t lo)
-
inline uint64_t lo_offset(uint64_t lo)
-
constexpr bool is_similarity_metric(MetricType metric_type)
this function is used to distinguish between min and max indexes since we need to support similarity and dis-similarity metrics in a flexible way
-
template<int A = 32>
inline bool is_aligned_pointer(const void *x)
-
inline uint16_t encode_bf16(const float f)
-
inline float decode_bf16(const uint16_t v)
-
float fvec_L2sqr(const float *x, const float *y, size_t d)
Squared L2 distance between two vectors.
-
float fvec_inner_product(const float *x, const float *y, size_t d)
inner product
-
float fvec_L1(const float *x, const float *y, size_t d)
L1 distance.
-
float fvec_Linf(const float *x, const float *y, size_t d)
infinity distance
-
void fvec_inner_product_batch_4(const float *x, const float *y0, const float *y1, const float *y2, const float *y3, const size_t d, float &dis0, float &dis1, float &dis2, float &dis3)
Special version of inner product that computes 4 distances between x and yi, which is performance oriented.
-
void fvec_L2sqr_batch_4(const float *x, const float *y0, const float *y1, const float *y2, const float *y3, const size_t d, float &dis0, float &dis1, float &dis2, float &dis3)
Special version of L2sqr that computes 4 distances between x and yi, which is performance oriented.
-
void pairwise_L2sqr(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)
Compute pairwise distances between sets of vectors
- Parameters:
d – dimension of the vectors
nq – nb of query vectors
nb – nb of database vectors
xq – query vectors (size nq * d)
xb – database vectors (size nb * d)
dis – output distances (size nq * nb)
ldq, ldb, ldd – strides for the matrices
-
void fvec_inner_products_ny(float *ip, const float *x, const float *y, size_t d, size_t ny)
-
void fvec_L2sqr_ny(float *dis, const float *x, const float *y, size_t d, size_t ny)
-
void fvec_L2sqr_ny_transposed(float *dis, const float *x, const float *y, const float *y_sqlen, size_t d, size_t d_offset, size_t ny)
-
size_t fvec_L2sqr_ny_nearest(float *distances_tmp_buffer, const float *x, const float *y, size_t d, size_t ny)
-
size_t fvec_L2sqr_ny_nearest_y_transposed(float *distances_tmp_buffer, const float *x, const float *y, const float *y_sqlen, size_t d, size_t d_offset, size_t ny)
-
float fvec_norm_L2sqr(const float *x, size_t d)
squared norm of a vector
-
void fvec_norms_L2(float *norms, const float *x, size_t d, size_t nx)
compute the L2 norms for a set of vectors
- Parameters:
norms – output norms, size nx
x – set of vectors, size nx * d
-
void fvec_norms_L2sqr(float *norms, const float *x, size_t d, size_t nx)
same as fvec_norms_L2, but computes squared norms
-
void fvec_renorm_L2(size_t d, size_t nx, float *x)
-
void inner_product_to_L2sqr(float *dis, const float *nr1, const float *nr2, size_t n1, size_t n2)
-
void fvec_add(size_t d, const float *a, const float *b, float *c)
compute c := a + b for vectors
c and a can overlap, c and b can overlap
- Parameters:
a – size d
b – size d
c – size d
-
void fvec_add(size_t d, const float *a, float b, float *c)
compute c := a + b for a, c vectors and b a scalar
c and a can overlap
- Parameters:
a – size d
c – size d
-
void fvec_sub(size_t d, const float *a, const float *b, float *c)
compute c := a - b for vectors
c and a can overlap, c and b can overlap
- Parameters:
a – size d
b – size d
c – size d
-
void fvec_inner_products_by_idx(float *ip, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)
compute the inner product between x and a subset y of ny vectors defined by ids
ip(i, j) = inner_product(x(i, :), y(ids(i, j), :))
- Parameters:
ip – output array, size nx * ny
x – first-term vector, size nx * d
y – second-term vector, size (max(ids) + 1) * d
ids – ids to sample from y, size nx * ny
-
void fvec_L2sqr_by_idx(float *dis, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)
compute the squared L2 distances between x and a subset y of ny vectors defined by ids
dis(i, j) = inner_product(x(i, :), y(ids(i, j), :))
- Parameters:
dis – output array, size nx * ny
x – first-term vector, size nx * d
y – second-term vector, size (max(ids) + 1) * d
ids – ids to sample from y, size nx * ny
-
void pairwise_indexed_L2sqr(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)
compute dis[j] = L2sqr(x[ix[j]], y[iy[j]]) forall j=0..n-1
- Parameters:
x – size (max(ix) + 1, d)
y – size (max(iy) + 1, d)
ix – size n
iy – size n
dis – size n
-
void pairwise_indexed_inner_product(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)
compute dis[j] = inner_product(x[ix[j]], y[iy[j]]) forall j=0..n-1
- Parameters:
x – size (max(ix) + 1, d)
y – size (max(iy) + 1, d)
ix – size n
iy – size n
dis – size n
-
void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_minheap_array_t *res, const IDSelector *sel = nullptr)
Return the k nearest neighbors of each of the nx vectors x among the ny vector y, w.r.t to max inner product.
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
res – result heap structure, which also provides k. Sorted on output
-
void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes, const IDSelector *sel = nullptr)
Return the k nearest neighbors of each of the nx vectors x among the ny vector y, for the inner product metric.
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
distances – output distances, size nq * k
indexes – output vector ids, size nq * k
-
void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_maxheap_array_t *res, const float *y_norm2 = nullptr, const IDSelector *sel = nullptr)
Return the k nearest neighbors of each of the nx vectors x among the ny vector y, for the L2 distance
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
res – result heap strcture, which also provides k. Sorted on output
y_norm2 – (optional) norms for the y vectors (nullptr or size ny)
sel – search in this subset of vectors
-
void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes, const float *y_norm2 = nullptr, const IDSelector *sel = nullptr)
Return the k nearest neighbors of each of the nx vectors x among the ny vector y, for the L2 distance
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
distances – output distances, size nq * k
indexes – output vector ids, size nq * k
y_norm2 – (optional) norms for the y vectors (nullptr or size ny)
sel – search in this subset of vectors
-
void knn_inner_products_by_idx(const float *x, const float *y, const int64_t *subset, size_t d, size_t nx, size_t ny, size_t nsubset, size_t k, float *vals, int64_t *ids, int64_t ld_ids = -1)
Find the max inner product neighbors for nx queries in a set of ny vectors indexed by ids. May be useful for re-ranking a pre-selected vector list
- Parameters:
x – query vectors, size nx * d
y – database vectors, size (max(ids) + 1) * d
ids – subset of database vectors to consider, size (nx, nsubset)
res – result structure
ld_ids – stride for the ids array. -1: use nsubset, 0: all queries process the same subset
-
void knn_L2sqr_by_idx(const float *x, const float *y, const int64_t *subset, size_t d, size_t nx, size_t ny, size_t nsubset, size_t k, float *vals, int64_t *ids, int64_t ld_subset = -1)
Find the nearest neighbors for nx queries in a set of ny vectors indexed by ids. May be useful for re-ranking a pre-selected vector list
- Parameters:
x – query vectors, size nx * d
y – database vectors, size (max(ids) + 1) * d
subset – subset of database vectors to consider, size (nx, nsubset)
res – rIDesult structure
ld_subset – stride for the subset array. -1: use nsubset, 0: all queries process the same subset
-
void range_search_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result, const IDSelector *sel = nullptr)
Return the k nearest neighbors of each of the nx vectors x among the ny vector y, w.r.t to max inner product
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
radius – search radius around the x vectors
result – result structure
-
void range_search_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result, const IDSelector *sel = nullptr)
same as range_search_L2sqr for the inner product similarity
-
void compute_PQ_dis_tables_dsub2(size_t d, size_t ksub, const float *centroids, size_t nx, const float *x, bool is_inner_product, float *dis_tables)
specialized function for PQ2
-
void fvec_madd(size_t n, const float *a, float bf, const float *b, float *c)
compute c := a + bf * b for a, b and c tables
- Parameters:
n – size of the tables
a – size n
b – size n
c – result table, size n
-
int fvec_madd_and_argmin(size_t n, const float *a, float bf, const float *b, float *c)
same as fvec_madd, also return index of the min of the result table
- Returns:
index of the min of table c
-
bool exhaustive_L2sqr_fused_cmax(const float *x, const float *y, size_t d, size_t nx, size_t ny, Top1BlockResultHandler<CMax<float, int64_t>> &res, const float *y_norms)
-
template<class Consumer, class ...Types>
Consumer::T dispatch_VectorDistance(size_t d, MetricType metric, float metric_arg, Consumer &consumer, Types... args)
-
void pairwise_extra_distances(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, MetricType mt, float metric_arg, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)
-
void knn_extra_metrics(const float *x, const float *y, size_t d, size_t nx, size_t ny, MetricType mt, float metric_arg, size_t k, float *distances, int64_t *indexes)
-
FlatCodesDistanceComputer *get_extra_distance_computer(size_t d, MetricType mt, float metric_arg, size_t nb, const float *xb)
get a DistanceComputer that refers to this type of distance and indexes a flat array of size nb
-
inline uint16_t encode_fp16(float x)
-
inline float decode_fp16(uint16_t x)
-
void bitvec_print(const uint8_t *b, size_t d)
-
void fvecs2bitvecs(const float *x, uint8_t *b, size_t d, size_t n)
-
void bitvecs2fvecs(const uint8_t *b, float *x, size_t d, size_t n)
-
void fvec2bitvec(const float *x, uint8_t *b, size_t d)
-
void bitvec_shuffle(size_t n, size_t da, size_t db, const int *order, const uint8_t *a, uint8_t *b)
Shuffle the bits from b(i, j) := a(i, order[j])
-
void hammings(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t nbytespercode, hamdis_t *dis)
Compute a set of Hamming distances between na and nb binary vectors
- Parameters:
a – size na * nbytespercode
b – size nb * nbytespercode
nbytespercode – should be multiple of 8
dis – output distances, size na * nb
-
void hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered, ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK)
Return the k smallest Hamming distances for a set of binary query vectors, using a max heap.
- Parameters:
a – queries, size ha->nh * ncodes
b – database, size nb * ncodes
nb – number of database vectors
ncodes – size of the binary codes (bytes)
ordered – if != 0: order the results by decreasing distance (may be bottleneck for k/n > 0.01)
approx_topk_mode – allows to use approximate top-k facilities to speedup heap
-
void hammings_knn(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered)
-
void hammings_knn_mc(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t k, size_t ncodes, int32_t *distances, int64_t *labels)
Return the k smallest Hamming distances for a set of binary query vectors, using counting max.
- Parameters:
a – queries, size na * ncodes
b – database, size nb * ncodes
na – number of query vectors
nb – number of database vectors
k – number of vectors/distances to return
ncodes – size of the binary codes (bytes)
distances – output distances from each query vector to its k nearest neighbors
labels – output ids of the k nearest neighbors to each query vector
-
void hamming_range_search(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, int radius, size_t ncodes, RangeSearchResult *result)
same as hammings_knn except we are doing a range search with radius
-
void hamming_count_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, size_t *nptr)
-
size_t match_hamming_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, int64_t *idx, hamdis_t *dis)
-
void crosshamming_count_thres(const uint8_t *dbs, size_t n, hamdis_t ht, size_t ncodes, size_t *nptr)
-
void generalized_hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t code_size, int ordered = true)
generalized Hamming distances (= count number of code bytes that are the same)
-
void pack_bitstrings(size_t n, size_t M, int nbit, const int32_t *unpacked, uint8_t *packed, size_t code_size)
Pack a set of n codes of size M * nbit
- Parameters:
n – number of codes to pack
M – number of elementary codes per code
nbit – number of bits per elementary code
unpacked – input unpacked codes, size (n, M)
packed – output packed codes, size (n, code_size)
code_size – should be >= ceil(M * nbit / 8)
-
void pack_bitstrings(size_t n, size_t M, const int32_t *nbits, const int32_t *unpacked, uint8_t *packed, size_t code_size)
Pack a set of n codes of variable sizes
- Parameters:
nbit – number of bits per entry (size M)
-
void unpack_bitstrings(size_t n, size_t M, int nbit, const uint8_t *packed, size_t code_size, int32_t *unpacked)
Unpack a set of n codes of size M * nbit
- Parameters:
n – number of codes to pack
M – number of elementary codes per code
nbit – number of bits per elementary code
unpacked – input unpacked codes, size (n, M)
packed – output packed codes, size (n, code_size)
code_size – should be >= ceil(M * nbit / 8)
-
void unpack_bitstrings(size_t n, size_t M, const int32_t *nbits, const uint8_t *packed, size_t code_size, int32_t *unpacked)
Unpack a set of n codes of variable sizes
- Parameters:
nbit – number of bits per entry (size M)
-
inline int generalized_hamming_64(uint64_t a)
-
inline int popcount32(uint32_t x)
-
inline int popcount64(uint64_t x)
- SPECIALIZED_HC (4)
- SPECIALIZED_HC (8)
- SPECIALIZED_HC (16)
- SPECIALIZED_HC (20)
- SPECIALIZED_HC (32)
- SPECIALIZED_HC (64)
-
template<class Consumer, class ...Types>
Consumer::T dispatch_HammingComputer(int code_size, Consumer &consumer, Types... args)
-
template<class C>
inline void heap_pop(size_t k, typename C::T *bh_val, typename C::TI *bh_ids) Pops the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1]. on output the element at k-1 is undefined.
-
template<class C>
inline void heap_push(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id) Pushes the element (val, ids) into the heap bh_val[0..k-2] and bh_ids[0..k-2]. on output the element at k-1 is defined.
-
template<class C>
inline void heap_replace_top(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id) Replaces the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1], and for identical bh_val[] values also sorts by bh_ids[] values.
-
template<typename T>
inline void minheap_pop(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline void minheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void minheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void maxheap_pop(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline void maxheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void maxheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<class C>
inline void heap_pop(size_t k, std::pair<typename C::T, typename C::TI> *bh) Pops the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1]. on output the element at k-1 is undefined.
-
template<class C>
inline void heap_push(size_t k, std::pair<typename C::T, typename C::TI> *bh, typename C::T val, typename C::TI id) Pushes the element (val, ids) into the heap bh_val[0..k-2] and bh_ids[0..k-2]. on output the element at k-1 is defined.
-
template<class C>
inline void heap_replace_top(size_t k, std::pair<typename C::T, typename C::TI> *bh, typename C::T val, typename C::TI id) Replaces the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1], and for identical bh_val[] values also sorts by bh_ids[] values.
-
template<class C>
inline void heap_heapify(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x = nullptr, const typename C::TI *ids = nullptr, size_t k0 = 0)
-
template<typename T>
inline void minheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)
-
template<typename T>
inline void maxheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)
-
template<class C>
inline void heap_addn(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x, const typename C::TI *ids, size_t n)
-
template<typename T>
inline void minheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)
-
template<typename T>
inline void maxheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)
-
template<typename C>
inline size_t heap_reorder(size_t k, typename C::T *bh_val, typename C::TI *bh_ids)
-
template<typename T>
inline size_t minheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline size_t maxheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)
-
template<class C>
inline void indirect_heap_pop(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids)
-
template<class C>
inline void indirect_heap_push(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids, typename C::TI id)
-
template<class idx_t, class C>
void merge_knn_results(size_t n, size_t k, typename C::TI nshard, const typename C::T *all_distances, const idx_t *all_labels, typename C::T *distances, idx_t *labels) Merge result tables from several shards. The per-shard results are assumed to be sorted. Note that the C comparator is reversed w.r.t. the usual top-k element heap because we want the best (ie. lowest for L2) result to be on top, not the worst. Also, it needs to hold an index of a shard id (ie. usually int32 is more than enough).
- Parameters:
all_distances – size (nshard, n, k)
all_labels – size (nshard, n, k)
distances – output distances, size (n, k)
labels – output labels, size (n, k)
-
template<>
inline float cmin_nextafter<float>(float x)
-
template<>
inline float cmax_nextafter<float>(float x)
-
template<>
inline uint16_t cmin_nextafter<uint16_t>(uint16_t x)
-
template<>
inline uint16_t cmax_nextafter<uint16_t>(uint16_t x)
-
template<class C>
C::T partition_fuzzy(typename C::T *vals, typename C::TI *ids, size_t n, size_t q_min, size_t q_max, size_t *q_out) partitions the table into 0:q and q:n where all elements above q are >= all elements below q (for C = CMax, for CMin comparisons are reversed)
Returns the partition threshold. The elements q:n are destroyed on output.
-
template<class C>
inline C::T partition(typename C::T *vals, typename C::TI *ids, size_t n, size_t q) simplified interface for when the parition is not fuzzy
-
void simd_histogram_8(const uint16_t *data, int n, uint16_t min, int shift, int *hist)
low level SIMD histogramming functions 8-bin histogram of (x - min) >> shift values outside the range are ignored. the data table should be aligned on 32 bytes
-
void simd_histogram_16(const uint16_t *data, int n, uint16_t min, int shift, int *hist)
same for 16-bin histogram
-
void float_rand(float *x, size_t n, int64_t seed)
-
void float_randn(float *x, size_t n, int64_t seed)
-
void int64_rand(int64_t *x, size_t n, int64_t seed)
-
void byte_rand(uint8_t *x, size_t n, int64_t seed)
-
void int64_rand_max(int64_t *x, size_t n, uint64_t max, int64_t seed)
-
void rand_perm(int *perm, size_t n, int64_t seed)
-
void rand_perm_splitmix64(int *perm, size_t n, int64_t seed)
-
void rand_smooth_vectors(size_t n, size_t d, float *x, int64_t seed)
-
inline simd16uint16 min(simd16uint16 a, simd16uint16 b)
-
inline simd16uint16 max(simd16uint16 a, simd16uint16 b)
-
inline simd16uint16 combine2x2(simd16uint16 a, simd16uint16 b)
-
inline uint32_t cmp_ge32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)
-
inline uint32_t cmp_le32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)
-
inline simd16uint16 hadd(const simd16uint16 &a, const simd16uint16 &b)
-
inline void cmplt_min_max_fast(const simd16uint16 candidateValues, const simd16uint16 candidateIndices, const simd16uint16 currentValues, const simd16uint16 currentIndices, simd16uint16 &minValues, simd16uint16 &minIndices, simd16uint16 &maxValues, simd16uint16 &maxIndices)
-
inline simd32uint8 uint16_to_uint8_saturate(simd16uint16 a, simd16uint16 b)
-
inline uint32_t get_MSBs(simd32uint8 a)
get most significant bit of each byte
-
inline simd32uint8 blendv(simd32uint8 a, simd32uint8 b, simd32uint8 mask)
use MSB of each byte of mask to select a byte between a and b
-
inline void cmplt_min_max_fast(const simd8uint32 candidateValues, const simd8uint32 candidateIndices, const simd8uint32 currentValues, const simd8uint32 currentIndices, simd8uint32 &minValues, simd8uint32 &minIndices, simd8uint32 &maxValues, simd8uint32 &maxIndices)
-
inline simd8float32 hadd(simd8float32 a, simd8float32 b)
-
inline simd8float32 unpacklo(simd8float32 a, simd8float32 b)
-
inline simd8float32 unpackhi(simd8float32 a, simd8float32 b)
-
inline simd8float32 fmadd(simd8float32 a, simd8float32 b, simd8float32 c)
-
inline void cmplt_and_blend_inplace(const simd8float32 candidateValues, const simd8uint32 candidateIndices, simd8float32 &lowestValues, simd8uint32 &lowestIndices)
-
inline void cmplt_min_max_fast(const simd8float32 candidateValues, const simd8uint32 candidateIndices, const simd8float32 currentValues, const simd8uint32 currentIndices, simd8float32 &minValues, simd8uint32 &minIndices, simd8float32 &maxValues, simd8uint32 &maxIndices)
-
inline simd16uint16 combine4x2(simd32uint16 a, simd32uint16 b)
-
inline simd16uint16 min(const simd16uint16 &av, const simd16uint16 &bv)
-
inline simd16uint16 max(const simd16uint16 &av, const simd16uint16 &bv)
-
inline simd16uint16 combine2x2(const simd16uint16 &a, const simd16uint16 &b)
-
inline uint32_t cmp_ge32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)
-
inline uint32_t cmp_le32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)
-
inline simd32uint8 uint16_to_uint8_saturate(const simd16uint16 &a, const simd16uint16 &b)
-
inline uint32_t get_MSBs(const simd32uint8 &a)
get most significant bit of each byte
-
inline simd32uint8 blendv(const simd32uint8 &a, const simd32uint8 &b, const simd32uint8 &mask)
use MSB of each byte of mask to select a byte between a and b
-
inline simd8float32 hadd(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 unpacklo(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 unpackhi(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 fmadd(const simd8float32 &a, const simd8float32 &b, const simd8float32 &c)
-
void fvec_argsort(size_t n, const float *vals, size_t *perm)
Indirect sort of a floating-point array
- Parameters:
n – size of the array
vals – array to sort, size n
perm – output: permutation of [0..n-1], st. vals[perm[i + 1]] >= vals[perm[i]]
-
void fvec_argsort_parallel(size_t n, const float *vals, size_t *perm)
Same as fvec_argsort, parallelized
-
void bucket_sort(size_t nval, const uint64_t *vals, uint64_t nbucket, int64_t *lims, int64_t *perm, int nt = 0)
Bucket sort of a list of values
- Parameters:
vals – values to sort, size nval, max value nbucket - 1
lims – output limits of buckets, size nbucket + 1
perm – output buckets, the elements of bucket i are in perm[lims[i]:lims[i + 1]]
nt – number of threads (0 = pure sequential code)
-
void matrix_bucket_sort_inplace(size_t nrow, size_t ncol, int32_t *vals, int32_t nbucket, int64_t *lims, int nt = 0)
in-place bucket sort (with attention to memory=>int32) on input the values are in a nrow * col matrix we want to store the row numbers in the output.
- Parameters:
vals – positive values to sort, size nrow * ncol, max value nbucket - 1
lims – output limits of buckets, size nbucket + 1
nt – number of threads (0 = pure sequential code)
-
void matrix_bucket_sort_inplace(size_t nrow, size_t ncol, int64_t *vals, int64_t nbucket, int64_t *lims, int nt = 0)
same with int64 elements
-
void hashtable_int64_to_int64_init(int log2_capacity, int64_t *tab)
Hashtable implementation for int64 -> int64 with external storage implemented for fast batch add and lookup.
tab is of size 2 * (1 << log2_capacity) n is the number of elements to add or search
adding several values in a same batch: an arbitrary one gets added in different batches: the newer batch overwrites. raises an exception if capacity is exhausted.
-
void hashtable_int64_to_int64_add(int log2_capacity, int64_t *tab, size_t n, const int64_t *keys, const int64_t *vals)
-
void hashtable_int64_to_int64_lookup(int log2_capacity, const int64_t *tab, size_t n, const int64_t *keys, int64_t *vals)
-
std::string get_compile_options()
get compile options
-
std::string get_version()
-
double getmillisecs()
ms elapsed since some arbitrary epoch
-
size_t get_mem_usage_kb()
get current RSS usage in kB
-
uint64_t get_cycles()
-
void reflection(const float *u, float *x, size_t n, size_t d, size_t nu)
-
void matrix_qr(int m, int n, float *a)
compute the Q of the QR decomposition for m > n
- Parameters:
a – size n * m: input matrix and output Q
-
void ranklist_handle_ties(int k, int64_t *idx, const float *dis)
distances are supposed to be sorted. Sorts indices with same distance
-
size_t ranklist_intersection_size(size_t k1, const int64_t *v1, size_t k2, const int64_t *v2)
count the number of common elements between v1 and v2 algorithm = sorting + bissection to avoid double-counting duplicates
-
size_t merge_result_table_with(size_t n, size_t k, int64_t *I0, float *D0, const int64_t *I1, const float *D1, bool keep_min = true, int64_t translation = 0)
merge a result table into another one
- Parameters:
I0, D0 – first result table, size (n, k)
I1, D1 – second result table, size (n, k)
keep_min – if true, keep min values, otherwise keep max
translation – add this value to all I1’s indexes
- Returns:
nb of values that were taken from the second table
-
double imbalance_factor(int n, int k, const int64_t *assign)
a balanced assignment has a IF of 1
-
double imbalance_factor(int k, const int *hist)
same, takes a histogram as input
-
int ivec_hist(size_t n, const int *v, int vmax, int *hist)
compute histogram on v
-
void bincode_hist(size_t n, size_t nbits, const uint8_t *codes, int *hist)
Compute histogram of bits on a code array
- Parameters:
codes – size(n, nbits / 8)
hist – size(nbits): nb of 1s in the array of codes
-
uint64_t ivec_checksum(size_t n, const int32_t *a)
compute a checksum on a table.
-
uint64_t bvec_checksum(size_t n, const uint8_t *a)
compute a checksum on a table.
-
void bvecs_checksum(size_t n, size_t d, const uint8_t *a, uint64_t *cs)
compute checksums for the rows of a matrix
- Parameters:
n – number of rows
d – size per row
a – matrix to handle, size n * d
cs – output checksums, size n
-
const float *fvecs_maybe_subsample(size_t d, size_t *n, size_t nmax, const float *x, bool verbose = false, int64_t seed = 1234)
random subsamples a set of vectors if there are too many of them
- Parameters:
d – dimension of the vectors
n – on input: nb of input vectors, output: nb of output vectors
nmax – max nb of vectors to keep
x – input array, size *n-by-d
seed – random seed to use for sampling
- Returns:
x or an array allocated with new [] with *n vectors
-
void binary_to_real(size_t d, const uint8_t *x_in, float *x_out)
Convert binary vector to +1/-1 valued float vector.
- Parameters:
d – dimension of the vector (multiple of 8)
x_in – input binary vector (uint8_t table of size d / 8)
x_out – output float vector (float table of size d)
-
void real_to_binary(size_t d, const float *x_in, uint8_t *x_out)
Convert float vector to binary vector. Components > 0 are converted to 1, others to 0.
- Parameters:
d – dimension of the vector (multiple of 8)
x_in – input float vector (float table of size d)
x_out – output binary vector (uint8_t table of size d / 8)
-
uint64_t hash_bytes(const uint8_t *bytes, int64_t n)
A reasonable hashing function
-
bool check_openmp()
Whether OpenMP annotations were respected.
Variables
- FAISS_API HNSWStats hnsw_stats
- FAISS_API int product_quantizer_compute_codes_bs
- FAISS_API int distance_compute_min_k_reservoir
- FAISS_API bool simd_result_handlers_accept_virtual
- FAISS_API int index2layer_sa_encode_bs
- FAISS_API int index_factory_verbose
set to > 0 to get more logs from index_factory
-
const int IO_FLAG_SKIP_STORAGE = 1
skip the storage for graph-based indexes
-
const int IO_FLAG_READ_ONLY = 2
-
const int IO_FLAG_ONDISK_SAME_DIR = 4
-
const int IO_FLAG_SKIP_IVF_DATA = 8
-
const int IO_FLAG_SKIP_PRECOMPUTE_TABLE = 16
-
const int IO_FLAG_PQ_SKIP_SDC_TABLE = 32
-
const int IO_FLAG_MMAP = IO_FLAG_SKIP_IVF_DATA | 0x646f0000
- FAISS_API IndexBinaryHashStats indexBinaryHash_stats
- FAISS_API FastScanStats FastScan_stats
- FAISS_API bool check_compatible_for_merge_expensive_check
- FAISS_API IndexIVFStats indexIVF_stats
- FAISS_API IVFFastScanStats IVFFastScan_stats
- FAISS_API size_t precomputed_table_max_bytes
- FAISS_API int index_ivfpq_add_core_o_bs
- FAISS_API IndexIVFPQStats indexIVFPQ_stats
- FAISS_API IndexPQStats indexPQ_stats
- FAISS_API int multi_index_quantizer_search_bs
- FAISS_API int rowwise_minmax_sa_encode_bs
block size for performing sa_encode and sa_decode
- FAISS_API int rowwise_minmax_sa_decode_bs
- FAISS_API int distance_compute_blas_threshold
- FAISS_API int distance_compute_blas_query_bs
- FAISS_API int distance_compute_blas_database_bs
- FAISS_API size_t hamming_batch_size
-
static constexpr uint8_t hamdis_tab_ham_bytes[256] = {0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8}
- FAISS_API PartitionStats partition_stats
- FAISS_API int bucket_sort_verbose
increase verbosity of the bucket_sort functions
-
struct AutoTuneCriterion
- #include <AutoTune.h>
Evaluation criterion. Returns a performance measure in [0,1], higher is better.
Subclassed by faiss::IntersectionCriterion, faiss::OneRecallAtRCriterion
Public Functions
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const = 0
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline virtual ~AutoTuneCriterion()
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
-
struct OneRecallAtRCriterion : public faiss::AutoTuneCriterion
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~OneRecallAtRCriterion() override
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const override
-
struct IntersectionCriterion : public faiss::AutoTuneCriterion
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~IntersectionCriterion() override
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const override
-
struct OperatingPoint
- #include <AutoTune.h>
Maintains a list of experimental results. Each operating point is a (perf, t, key) triplet, where higher perf and lower t is better. The key field is an arbitrary identifier for the operating point.
Includes primitives to extract the Pareto-optimal operating points in the (perf, t) space.
Public Members
-
double perf
performance measure (output of a Criterion)
-
double t
corresponding execution time (ms)
-
std::string key
key that identifies this op pt
-
int64_t cno
integer identifier
-
double perf
-
struct OperatingPoints
Public Functions
-
OperatingPoints()
-
int merge_with(const OperatingPoints &other, const std::string &prefix = "")
add operating points from other to this, with a prefix to the keys
-
void clear()
-
bool add(double perf, double t, const std::string &key, size_t cno = 0)
add a performance measure. Return whether it is an optimal point
-
double t_for_perf(double perf) const
get time required to obtain a given performance measure
-
void display(bool only_optimal = true) const
easy-to-read output
-
void all_to_gnuplot(const char *fname) const
output to a format easy to digest by gnuplot
-
void optimal_to_gnuplot(const char *fname) const
Public Members
-
std::vector<OperatingPoint> all_pts
all operating points
-
std::vector<OperatingPoint> optimal_pts
optimal operating points, sorted by perf
-
OperatingPoints()
-
struct ParameterRange
- #include <AutoTune.h>
possible values of a parameter, sorted from least to most expensive/accurate
-
struct ParameterSpace
- #include <AutoTune.h>
Uses a-priori knowledge on the Faiss indexes to extract tunable parameters.
Subclassed by faiss::gpu::GpuParameterSpace
Public Functions
-
ParameterSpace()
-
size_t n_combinations() const
nb of combinations, = product of values sizes
-
bool combination_ge(size_t c1, size_t c2) const
returns whether combinations c1 >= c2 in the tuple sense
-
std::string combination_name(size_t cno) const
get string representation of the combination
-
void display() const
print a description on stdout
-
ParameterRange &add_range(const std::string &name)
add a new parameter (or return it if it exists)
-
virtual void initialize(const Index *index)
initialize with reasonable parameters for the index
-
void set_index_parameters(Index *index, size_t cno) const
set a combination of parameters on an index
-
void set_index_parameters(Index *index, const char *param_string) const
set a combination of parameters described by a string
-
virtual void set_index_parameter(Index *index, const std::string &name, double val) const
set one of the parameters, returns whether setting was successful
-
void update_bounds(size_t cno, const OperatingPoint &op, double *upper_bound_perf, double *lower_bound_t) const
find an upper bound on the performance and a lower bound on t for configuration cno given another operating point op
-
void explore(Index *index, size_t nq, const float *xq, const AutoTuneCriterion &crit, OperatingPoints *ops) const
explore operating points
- Parameters:
index – index to run on
xq – query vectors (size nq * index.d)
crit – selection criterion
ops – resulting operating points
-
inline virtual ~ParameterSpace()
Public Members
-
std::vector<ParameterRange> parameter_ranges
all tunable parameters
-
int verbose
verbosity during exploration
-
int n_experiments
nb of experiments during optimization (0 = try all combinations)
-
size_t batchsize
maximum number of queries to submit at a time.
-
bool thread_over_batches
use multithreading over batches (useful to benchmark independent single-searches)
-
double min_test_duration
run tests several times until they reach at least this duration (to avoid jittering in MT mode)
-
ParameterSpace()
-
struct Cloner
- #include <clone_index.h>
Cloner class, useful to override classes with other cloning functions. The cloning function above just calls Cloner::clone_Index.
Subclassed by faiss::gpu::ToCPUCloner, faiss::gpu::ToGpuCloner, faiss::gpu::ToGpuClonerMultiple
Public Functions
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
inline virtual ~Cloner()
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
struct ClusteringParameters
- #include <Clustering.h>
Class for the clustering parameters. Can be passed to the constructor of the Clustering object.
Subclassed by faiss::Clustering, faiss::ProgressiveDimClusteringParameters
Public Members
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
bool check_input_data_for_NaNs = true
whether to check for NaNs in an input data
-
bool use_faster_subsampling = false
Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.
-
int niter = 25
-
struct ClusteringIterationStats
Public Members
-
float obj
objective values (sum of distances reported by index)
-
double time
seconds for iteration
-
double time_search
seconds for just search
-
double imbalance_factor
imbalance factor of iteration
-
int nsplit
number of cluster splits
-
float obj
-
struct Clustering : public faiss::ClusteringParameters
- #include <Clustering.h>
K-means clustering based on assignment - centroid update iterations
The clustering is based on an Index object that assigns training points to the centroids. Therefore, at each iteration the centroids are added to the index.
On output, the centoids table is set to the latest version of the centroids and they are also added to the index. If the centroids table it is not empty on input, it is also used for initialization.
Subclassed by faiss::Clustering1D
Public Functions
-
Clustering(int d, int k)
-
Clustering(int d, int k, const ClusteringParameters &cp)
-
virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)
run k-means training
- Parameters:
x – training vectors, size n * d
index – index used for assignment
x_weights – weight associated to each vector: NULL or size n
-
void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)
run with encoded vectors
win addition to train()’s parameters takes a codec as parameter to decode the input vectors.
- Parameters:
codec – codec used to decode the vectors (nullptr = vectors are in fact floats)
-
void post_process_centroids()
Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding
-
inline virtual ~Clustering()
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d) if centroids are set on input to train, they will be used as initialization
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
bool check_input_data_for_NaNs = true
whether to check for NaNs in an input data
-
bool use_faster_subsampling = false
Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.
-
Clustering(int d, int k)
-
struct Clustering1D : public faiss::Clustering
- #include <Clustering.h>
Exact 1D clustering algorithm
Since it does not use an index, it does not overload the train() function
Public Functions
-
explicit Clustering1D(int k)
-
Clustering1D(int k, const ClusteringParameters &cp)
-
void train_exact(idx_t n, const float *x)
-
inline virtual ~Clustering1D()
-
virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)
run k-means training
- Parameters:
x – training vectors, size n * d
index – index used for assignment
x_weights – weight associated to each vector: NULL or size n
-
void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)
run with encoded vectors
win addition to train()’s parameters takes a codec as parameter to decode the input vectors.
- Parameters:
codec – codec used to decode the vectors (nullptr = vectors are in fact floats)
-
void post_process_centroids()
Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d) if centroids are set on input to train, they will be used as initialization
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
bool check_input_data_for_NaNs = true
whether to check for NaNs in an input data
-
bool use_faster_subsampling = false
Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.
-
explicit Clustering1D(int k)
-
struct ProgressiveDimClusteringParameters : public faiss::ClusteringParameters
Subclassed by faiss::ProgressiveDimClustering
Public Functions
-
ProgressiveDimClusteringParameters()
Public Members
-
int progressive_dim_steps
number of incremental steps
-
bool apply_pca
apply PCA on input
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
bool check_input_data_for_NaNs = true
whether to check for NaNs in an input data
-
bool use_faster_subsampling = false
Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.
-
ProgressiveDimClusteringParameters()
-
struct ProgressiveDimIndexFactory
- #include <Clustering.h>
generates an index suitable for clustering when called
Subclassed by faiss::gpu::GpuProgressiveDimIndexFactory
Public Functions
-
virtual Index *operator()(int dim)
ownership transferred to caller
-
inline virtual ~ProgressiveDimIndexFactory()
-
virtual Index *operator()(int dim)
-
struct ProgressiveDimClustering : public faiss::ProgressiveDimClusteringParameters
- #include <Clustering.h>
K-means clustering with progressive dimensions used
The clustering first happens in dim 1, then with exponentially increasing dimension until d (I steps). This is typically applied after a PCA transformation (optional). Reference:
“Improved Residual Vector Quantization for High-dimensional Approximate
Nearest Neighbor Search”
Shicong Liu, Hongtao Lu, Junru Shao, AAAI’15
https://arxiv.org/abs/1509.05195
Public Functions
-
ProgressiveDimClustering(int d, int k)
-
ProgressiveDimClustering(int d, int k, const ProgressiveDimClusteringParameters &cp)
-
void train(idx_t n, const float *x, ProgressiveDimIndexFactory &factory)
-
inline virtual ~ProgressiveDimClustering()
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d)
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int progressive_dim_steps
number of incremental steps
-
bool apply_pca
apply PCA on input
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator. negative values lead to seeding an internal rng with std::high_resolution_clock.
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
bool check_input_data_for_NaNs = true
whether to check for NaNs in an input data
-
bool use_faster_subsampling = false
Whether to use splitmix64-based random number generator for subsampling, which is faster, but may pick duplicate points.
-
ProgressiveDimClustering(int d, int k)
-
struct AdditiveQuantizer : public faiss::Quantizer
- #include <AdditiveQuantizer.h>
Abstract structure for additive quantizers
Different from the product quantizer in which the decoded vector is the concatenation of M sub-vectors, additive quantizers sum M sub-vectors to get the decoded vector.
Subclassed by faiss::LocalSearchQuantizer, faiss::ProductAdditiveQuantizer, faiss::ResidualQuantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
void compute_codebook_tables()
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
AdditiveQuantizer(size_t d, const std::vector<size_t> &nbits, Search_type_t search_type = ST_decompress)
-
AdditiveQuantizer()
compute derived values when d, M and nbits have been set
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const = 0
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void decode_64bit(idx_t n, float *x) const
decoding function for a code in a 64-bit word
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const
Compute inner-product look-up tables. Used in the centroid search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
-
void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const
Exact L2 search, with precomputed norms
-
virtual ~AdditiveQuantizer()
-
virtual void train(size_t n, const float *x) = 0
Train the quantizer
- Parameters:
x – training vectors, size n * d
Public Members
-
size_t M
number of codebooks
-
std::vector<size_t> nbits
bits for each step
-
std::vector<float> codebooks
codebooks
-
std::vector<uint64_t> codebook_offsets
codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
std::vector<float> norm_tabs
auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan
-
IndexFlat1D qnorm
store and search norms
-
std::vector<float> centroid_norms
norms of all codebook entries (size total_codebook_size)
-
std::vector<float> codebook_cross_products
dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct RangeSearchResult
- #include <AuxIndexStructures.h>
The objective is to have a simple result structure while minimizing the number of mem copies in the result. The method do_allocation can be overloaded to allocate the result tables in the matrix type of a scripting language like Lua or Python.
Public Functions
-
explicit RangeSearchResult(size_t nq, bool alloc_lims = true)
lims must be allocated on input to range_search.
-
virtual void do_allocation()
called when lims contains the nb of elements result entries for each query
-
virtual ~RangeSearchResult()
Public Members
-
size_t nq
nb of queries
-
size_t *lims
size (nq + 1)
-
idx_t *labels
result for query i is labels[lims[i]:lims[i+1]]
-
float *distances
corresponding distances (not sorted)
-
size_t buffer_size
size of the result buffers used
-
explicit RangeSearchResult(size_t nq, bool alloc_lims = true)
-
struct BufferList
- #include <AuxIndexStructures.h>
List of temporary buffers used to store results before they are copied to the RangeSearchResult object.
Subclassed by faiss::RangeSearchPartialResult
Public Functions
-
explicit BufferList(size_t buffer_size)
-
~BufferList()
-
void append_buffer()
create a new buffer
-
void add(idx_t id, float dis)
add one result, possibly appending a new buffer if needed
-
void copy_range(size_t ofs, size_t n, idx_t *dest_ids, float *dest_dis)
copy elemnts ofs:ofs+n-1 seen as linear data in the buffers to tables dest_ids, dest_dis
Public Members
-
size_t buffer_size
-
size_t wp
write pointer in the last buffer.
-
struct Buffer
-
explicit BufferList(size_t buffer_size)
-
struct RangeQueryResult
- #include <AuxIndexStructures.h>
result structure for a single query
Public Functions
-
void add(float dis, idx_t id)
called by search function to report a new result
-
void add(float dis, idx_t id)
-
struct RangeSearchPartialResult : public faiss::BufferList
- #include <AuxIndexStructures.h>
the entries in the buffers are split per query
Public Functions
-
explicit RangeSearchPartialResult(RangeSearchResult *res_in)
eventually the result will be stored in res_in
-
RangeQueryResult &new_result(idx_t qno)
begin a new result
-
void finalize()
-
void set_lims()
called by range_search before do_allocation
-
void copy_result(bool incremental = false)
called by range_search after do_allocation
-
void append_buffer()
create a new buffer
-
void add(idx_t id, float dis)
add one result, possibly appending a new buffer if needed
-
void copy_range(size_t ofs, size_t n, idx_t *dest_ids, float *dest_dis)
copy elemnts ofs:ofs+n-1 seen as linear data in the buffers to tables dest_ids, dest_dis
Public Members
-
RangeSearchResult *res
-
std::vector<RangeQueryResult> queries
query ids + nb of results per query.
-
size_t buffer_size
-
std::vector<Buffer> buffers
-
size_t wp
write pointer in the last buffer.
Public Static Functions
-
static void merge(std::vector<RangeSearchPartialResult*> &partial_results, bool do_delete = true)
merge a set of PartialResult’s into one RangeSearchResult on output the partialresults are empty!
-
explicit RangeSearchPartialResult(RangeSearchResult *res_in)
-
struct InterruptCallback
Subclassed by faiss::TimeoutCallback
Public Functions
-
virtual bool want_interrupt() = 0
-
inline virtual ~InterruptCallback()
Public Static Functions
-
static void clear_instance()
-
static void check()
check if:
an interrupt callback is set
the callback returns true if this is the case, then throw an exception. Should not be called from multiple threads.
-
static bool is_interrupted()
same as check() but return true if is interrupted instead of throwing. Can be called from multiple threads.
-
static size_t get_period_hint(size_t flops)
assuming each iteration takes a certain number of flops, what is a reasonable interval to check for interrupts?
-
virtual bool want_interrupt() = 0
-
struct TimeoutCallback : public faiss::InterruptCallback
Public Functions
-
virtual bool want_interrupt() override
-
void set_timeout(double timeout_in_seconds)
Public Static Functions
-
static void reset(double timeout_in_seconds)
-
static void clear_instance()
-
static void check()
check if:
an interrupt callback is set
the callback returns true if this is the case, then throw an exception. Should not be called from multiple threads.
-
static bool is_interrupted()
same as check() but return true if is interrupted instead of throwing. Can be called from multiple threads.
-
static size_t get_period_hint(size_t flops)
assuming each iteration takes a certain number of flops, what is a reasonable interval to check for interrupts?
-
virtual bool want_interrupt() override
-
struct VisitedTable
- #include <AuxIndexStructures.h>
set implementation optimized for fast access.
Public Functions
-
inline explicit VisitedTable(int size)
-
inline void set(int no)
set flag #no to true
-
inline bool get(int no) const
get flag #no
-
inline void advance()
reset all flags to false
-
inline explicit VisitedTable(int size)
-
struct CodePacker
- #include <CodePacker.h>
Packing consists in combining a fixed number of codes of constant size (code_size) into a block of data where they may (or may not) be interleaved for efficient consumption by distance computation kernels. This exists for the “fast_scan” indexes on CPU and for some GPU kernels.
Subclassed by faiss::CodePackerFlat, faiss::CodePackerPQ4
Public Functions
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const = 0
-
virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const = 0
-
virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const
-
virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const
-
inline virtual ~CodePacker()
Public Members
-
size_t code_size
-
size_t nvec
-
size_t block_size
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const = 0
-
struct CodePackerFlat : public faiss::CodePacker
- #include <CodePacker.h>
Trivial code packer where codes are stored one by one
Public Functions
-
explicit CodePackerFlat(size_t code_size)
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const final
-
virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const final
-
virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const final
-
virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const final
Public Members
-
size_t code_size
-
size_t nvec
-
size_t block_size
-
explicit CodePackerFlat(size_t code_size)
-
struct DistanceComputer
Subclassed by faiss::FlatCodesDistanceComputer, faiss::NegativeDistanceComputer
Public Functions
-
virtual void set_query(const float *x) = 0
called before computing distances. Pointer x should remain valid while operator () is called
-
virtual float operator()(idx_t i) = 0
compute distance of vector i to current query
-
inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3)
compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.
-
inline virtual ~DistanceComputer()
-
virtual void set_query(const float *x) = 0
-
struct NegativeDistanceComputer : public faiss::DistanceComputer
Public Functions
-
inline explicit NegativeDistanceComputer(DistanceComputer *basedis)
-
inline virtual void set_query(const float *x) override
called before computing distances. Pointer x should remain valid while operator () is called
-
inline virtual float operator()(idx_t i) override
compute distance of vector i to current query
-
inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3) override
compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.
-
inline virtual float symmetric_dis(idx_t i, idx_t j) override
compute distance between two stored vectors
-
inline virtual ~NegativeDistanceComputer()
Public Members
-
DistanceComputer *basedis
owned by this
-
inline explicit NegativeDistanceComputer(DistanceComputer *basedis)
-
struct FlatCodesDistanceComputer : public faiss::DistanceComputer
Subclassed by faiss::ScalarQuantizer::SQDistanceComputer
Public Functions
-
inline FlatCodesDistanceComputer(const uint8_t *codes, size_t code_size)
-
inline FlatCodesDistanceComputer()
-
inline virtual float operator()(idx_t i) override
compute distance of vector i to current query
-
virtual float distance_to_code(const uint8_t *code) = 0
compute distance of current query to an encoded vector
-
inline virtual ~FlatCodesDistanceComputer()
-
virtual void set_query(const float *x) = 0
called before computing distances. Pointer x should remain valid while operator () is called
-
inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3)
compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.
Public Members
-
const uint8_t *codes
-
size_t code_size
-
inline FlatCodesDistanceComputer(const uint8_t *codes, size_t code_size)
-
class FaissException : public std::exception
- #include <FaissException.h>
Base class for Faiss exceptions.
Public Functions
-
explicit FaissException(const std::string &msg)
-
FaissException(const std::string &msg, const char *funcName, const char *file, int line)
-
const char *what() const noexcept override
from std::exception
Public Members
-
std::string msg
-
explicit FaissException(const std::string &msg)
-
struct TransformedVectors
- #include <FaissException.h>
RAII object for a set of possibly transformed vectors (deallocated only if they are indeed transformed)
Public Functions
-
inline TransformedVectors(const float *x_orig, const float *x)
-
inline ~TransformedVectors()
Public Members
-
const float *x
-
bool own_x
-
inline TransformedVectors(const float *x_orig, const float *x)
-
template<class C>
struct ResultHandler Subclassed by faiss::HeapBlockResultHandler< C, use_sel >::SingleResultHandler, faiss::RangeSearchBlockResultHandler< C, use_sel >::SingleResultHandler, faiss::ReservoirTopN< C >, faiss::Top1BlockResultHandler< C, use_sel >::SingleResultHandler
Public Functions
-
inline virtual ~ResultHandler()
-
inline virtual ~ResultHandler()
-
struct SearchParametersHNSW : public faiss::SearchParameters
Public Functions
-
inline ~SearchParametersHNSW()
Public Members
-
int efSearch = 16
-
bool check_relative_distance = true
-
bool bounded_queue = true
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline ~SearchParametersHNSW()
-
struct HNSW
Public Types
-
using storage_idx_t = int32_t
internal storage of vectors (32 bits: this is expensive)
-
using C = CMax<float, int64_t>
-
typedef std::pair<float, storage_idx_t> Node
Public Functions
-
void set_default_probas(int M, float levelMult)
initialize the assign_probas and cum_nneighbor_per_level to have 2*M links on level 0 and M links on levels > 0
-
void set_nb_neighbors(int level_no, int n)
set nb of neighbors for this level (before adding anything)
-
int nb_neighbors(int layer_no) const
nb of neighbors for this level
-
int cum_nb_neighbors(int layer_no) const
cumumlative nb up to (and excluding) this level
-
void neighbor_range(idx_t no, int layer_no, size_t *begin, size_t *end) const
range of entries in the neighbors table of vertex no at layer_no
-
explicit HNSW(int M = 32)
only mandatory parameter: nb of neighbors
-
int random_level()
pick a random level for a new point
-
void fill_with_random_links(size_t n)
add n random levels to table (for debugging…)
-
void add_links_starting_from(DistanceComputer &ptdis, storage_idx_t pt_id, storage_idx_t nearest, float d_nearest, int level, omp_lock_t *locks, VisitedTable &vt, bool keep_max_size_level0 = false)
-
void add_with_locks(DistanceComputer &ptdis, int pt_level, int pt_id, std::vector<omp_lock_t> &locks, VisitedTable &vt, bool keep_max_size_level0 = false)
add point pt_id on all levels <= pt_level and build the link structure for them.
-
HNSWStats search(DistanceComputer &qdis, ResultHandler<C> &res, VisitedTable &vt, const SearchParametersHNSW *params = nullptr) const
search interface for 1 point, single thread
-
void search_level_0(DistanceComputer &qdis, ResultHandler<C> &res, idx_t nprobe, const storage_idx_t *nearest_i, const float *nearest_d, int search_type, HNSWStats &search_stats, VisitedTable &vt, const SearchParametersHNSW *params = nullptr) const
search only in level 0 from a given vertex
-
void reset()
-
void clear_neighbor_tables(int level)
-
void print_neighbor_stats(int level) const
-
int prepare_level_tab(size_t n, bool preset_levels = false)
-
void permute_entries(const idx_t *map)
Public Members
-
std::vector<double> assign_probas
assignment probability to each layer (sum=1)
-
std::vector<int> cum_nneighbor_per_level
number of neighbors stored per layer (cumulative), should not be changed after first add
-
std::vector<int> levels
level of each vector (base level = 1), size = ntotal
-
std::vector<size_t> offsets
offsets[i] is the offset in the neighbors array where vector i is stored size ntotal + 1
-
std::vector<storage_idx_t> neighbors
neighbors[offsets[i]:offsets[i+1]] is the list of neighbors of vector i for all levels. this is where all storage goes.
-
storage_idx_t entry_point = -1
entry point in the search structure (one of the points with maximum level
-
faiss::RandomGenerator rng
-
int max_level = -1
maximum level
-
int efConstruction = 40
expansion factor at construction time
-
int efSearch = 16
expansion factor at search time
-
bool check_relative_distance = true
during search: do we check whether the next best distance is good enough?
-
bool search_bounded_queue = true
use bounded queue during exploration
Public Static Functions
-
static void shrink_neighbor_list(DistanceComputer &qdis, std::priority_queue<NodeDistFarther> &input, std::vector<NodeDistFarther> &output, int max_size, bool keep_max_size_level0 = false)
-
struct MinimaxHeap
- #include <HNSW.h>
Heap structure that allows fast
Public Types
-
typedef faiss::CMax<float, storage_idx_t> HC
Public Functions
-
inline explicit MinimaxHeap(int n)
-
void push(storage_idx_t i, float v)
-
float max() const
-
int size() const
-
void clear()
-
int pop_min(float *vmin_out = nullptr)
-
int count_below(float thresh)
-
typedef faiss::CMax<float, storage_idx_t> HC
-
struct NodeDistCloser
- #include <HNSW.h>
to sort pairs of (id, distance) from nearest to fathest or the reverse
Public Functions
-
inline NodeDistCloser(float d, int id)
-
inline bool operator<(const NodeDistCloser &obj1) const
Public Members
-
float d
-
int id
-
inline NodeDistCloser(float d, int id)
-
struct NodeDistFarther
Public Functions
-
inline NodeDistFarther(float d, int id)
-
inline bool operator<(const NodeDistFarther &obj1) const
Public Members
-
float d
-
int id
-
inline NodeDistFarther(float d, int id)
-
using storage_idx_t = int32_t
-
struct HNSWStats
Public Functions
-
inline void reset()
number of hops aka number of edges traversed
-
inline void combine(const HNSWStats &other)
Public Members
-
size_t n1 = 0
-
size_t n2 = 0
number of vectors searched
-
size_t ndis = 0
number of queries for which the candidate list is exhausted
-
size_t nhops = 0
number of distances computed
-
inline void reset()
-
struct IDSelector
- #include <IDSelector.h>
Encapsulates a set of ids to handle.
Subclassed by PyCallbackIDSelector, faiss::IDSelectorAll, faiss::IDSelectorAnd, faiss::IDSelectorArray, faiss::IDSelectorBatch, faiss::IDSelectorBitmap, faiss::IDSelectorNot, faiss::IDSelectorOr, faiss::IDSelectorRange, faiss::IDSelectorTranslated, faiss::IDSelectorXOr
-
struct IDSelectorRange : public faiss::IDSelector
- #include <IDSelector.h>
ids between [imin, imax)
-
struct IDSelectorArray : public faiss::IDSelector
- #include <IDSelector.h>
Simple array of elements
is_member calls are very inefficient, but some operations can use the ids directly.
Public Functions
-
IDSelectorArray(size_t n, const idx_t *ids)
Construct with an array of ids to process
- Parameters:
n – number of ids to store
ids – elements to store. The pointer should remain valid during IDSelectorArray’s lifetime
-
virtual bool is_member(idx_t id) const final
-
inline ~IDSelectorArray() override
-
IDSelectorArray(size_t n, const idx_t *ids)
-
struct IDSelectorBatch : public faiss::IDSelector
- #include <IDSelector.h>
Ids from a set.
Repetitions of ids in the indices set passed to the constructor does not hurt performance.
The hash function used for the bloom filter and GCC’s implementation of unordered_set are just the least significant bits of the id. This works fine for random ids or ids in sequences but will produce many hash collisions if lsb’s are always the same
-
struct IDSelectorBitmap : public faiss::IDSelector
- #include <IDSelector.h>
One bit per element. Constructed with a bitmap, size ceil(n / 8).
Public Functions
-
IDSelectorBitmap(size_t n, const uint8_t *bitmap)
Construct with a binary mask
- Parameters:
n – size of the bitmap array
bitmap – id will be selected iff id / 8 < n and bit number (i%8) of bitmap[floor(i / 8)] is 1.
-
virtual bool is_member(idx_t id) const final
-
inline ~IDSelectorBitmap() override
Public Members
-
size_t n
-
const uint8_t *bitmap
-
IDSelectorBitmap(size_t n, const uint8_t *bitmap)
-
struct IDSelectorNot : public faiss::IDSelector
- #include <IDSelector.h>
reverts the membership test of another selector
Public Functions
-
inline IDSelectorNot(const IDSelector *sel)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorNot()
Public Members
-
const IDSelector *sel
-
inline IDSelectorNot(const IDSelector *sel)
-
struct IDSelectorAll : public faiss::IDSelector
- #include <IDSelector.h>
selects all entries (useful for benchmarking)
Public Functions
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorAll()
-
inline virtual bool is_member(idx_t id) const final
-
struct IDSelectorAnd : public faiss::IDSelector
- #include <IDSelector.h>
does an AND operation on the the two given IDSelector’s is_membership results.
Public Functions
-
inline IDSelectorAnd(const IDSelector *lhs, const IDSelector *rhs)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorAnd()
-
inline IDSelectorAnd(const IDSelector *lhs, const IDSelector *rhs)
-
struct IDSelectorOr : public faiss::IDSelector
- #include <IDSelector.h>
does an OR operation on the the two given IDSelector’s is_membership results.
Public Functions
-
inline IDSelectorOr(const IDSelector *lhs, const IDSelector *rhs)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorOr()
-
inline IDSelectorOr(const IDSelector *lhs, const IDSelector *rhs)
-
struct IDSelectorXOr : public faiss::IDSelector
- #include <IDSelector.h>
does an XOR operation on the the two given IDSelector’s is_membership results.
Public Functions
-
inline IDSelectorXOr(const IDSelector *lhs, const IDSelector *rhs)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorXOr()
-
inline IDSelectorXOr(const IDSelector *lhs, const IDSelector *rhs)
-
struct IOReader
Subclassed by PyCallbackIOReader, faiss::BufferedIOReader, faiss::FileIOReader, faiss::VectorIOReader
Public Functions
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) = 0
-
virtual int filedescriptor()
-
inline virtual ~IOReader()
Public Members
-
std::string name
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) = 0
-
struct IOWriter
Subclassed by PyCallbackIOWriter, faiss::BufferedIOWriter, faiss::FileIOWriter, faiss::VectorIOWriter
Public Functions
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) = 0
-
virtual int filedescriptor()
-
inline virtual ~IOWriter() noexcept(false)
Public Members
-
std::string name
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) = 0
-
struct VectorIOReader : public faiss::IOReader
Public Functions
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor()
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
struct VectorIOWriter : public faiss::IOWriter
Public Functions
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor()
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
struct FileIOReader : public faiss::IOReader
Public Functions
-
FileIOReader(FILE *rf)
-
FileIOReader(const char *fname)
-
~FileIOReader() override
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor() override
-
FileIOReader(FILE *rf)
-
struct FileIOWriter : public faiss::IOWriter
Public Functions
-
FileIOWriter(FILE *wf)
-
FileIOWriter(const char *fname)
-
~FileIOWriter() override
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor() override
-
FileIOWriter(FILE *wf)
-
struct BufferedIOReader : public faiss::IOReader
- #include <io.h>
wraps an ioreader to make buffered reads to avoid too small reads
Public Functions
-
explicit BufferedIOReader(IOReader *reader, size_t bsz = 1024 * 1024)
- Parameters:
bsz – buffer size (bytes). Reads will be done by batched of this size
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor()
-
explicit BufferedIOReader(IOReader *reader, size_t bsz = 1024 * 1024)
-
struct BufferedIOWriter : public faiss::IOWriter
Public Functions
-
explicit BufferedIOWriter(IOWriter *writer, size_t bsz = 1024 * 1024)
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
~BufferedIOWriter() override
-
virtual int filedescriptor()
-
explicit BufferedIOWriter(IOWriter *writer, size_t bsz = 1024 * 1024)
-
struct ZnSphereSearch
- #include <lattice_Zn.h>
returns the nearest vertex in the sphere to a query. Returns only the coordinates, not an id.
Algorithm: all points are derived from a one atom vector up to a permutation and sign changes. The search function finds the most appropriate atom and transformation.
Subclassed by faiss::ZnSphereCodec
Public Functions
-
ZnSphereSearch(int dim, int r2)
-
float search(const float *x, float *c) const
find nearest centroid. x does not need to be normalized
-
float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const
full call. Requires externally-allocated temp space
-
void search_multi(int n, const float *x, float *c_out, float *dp_out)
-
ZnSphereSearch(int dim, int r2)
-
struct EnumeratedVectors
Subclassed by faiss::ZnSphereCodec, faiss::ZnSphereCodecRec
Public Functions
-
inline explicit EnumeratedVectors(int dim)
-
virtual uint64_t encode(const float *x) const = 0
encode a vector from a collection
-
virtual void decode(uint64_t code, float *c) const = 0
decode it
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
-
inline virtual ~EnumeratedVectors()
Public Members
-
uint64_t nv
size of the collection
-
int dim
-
inline explicit EnumeratedVectors(int dim)
-
struct Repeat
Public Members
-
float val
-
int n
-
float val
-
struct Repeats
- #include <lattice_Zn.h>
Repeats: used to encode a vector that has n occurrences of val. Encodes the signs and permutation of the vector. Useful for atoms.
Subclassed by faiss::ZnSphereCodec::CodeSegment
Public Functions
-
Repeats(int dim = 0, const float *c = nullptr)
-
uint64_t count() const
-
uint64_t encode(const float *c) const
-
void decode(uint64_t code, float *c) const
-
Repeats(int dim = 0, const float *c = nullptr)
-
struct ZnSphereCodec : public faiss::ZnSphereSearch, public faiss::EnumeratedVectors
- #include <lattice_Zn.h>
codec that can return ids for the encoded vectors
uses the ZnSphereSearch to encode the vector by encoding the permutation and signs. Depends on ZnSphereSearch because it uses the atom numbers
Subclassed by faiss::ZnSphereCodecAlt
Public Functions
-
ZnSphereCodec(int dim, int r2)
-
uint64_t search_and_encode(const float *x) const
-
virtual void decode(uint64_t code, float *c) const override
decode it
-
virtual uint64_t encode(const float *x) const override
takes vectors that do not need to be centroids
-
float search(const float *x, float *c) const
find nearest centroid. x does not need to be normalized
-
float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const
full call. Requires externally-allocated temp space
-
void search_multi(int n, const float *x, float *c_out, float *dp_out)
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
Public Members
-
std::vector<CodeSegment> code_segments
-
uint64_t nv
-
size_t code_size
-
int dimS
-
int r2
-
int natom
-
std::vector<float> voc
size dim * ntatom
-
int dim
-
ZnSphereCodec(int dim, int r2)
-
struct ZnSphereCodecRec : public faiss::EnumeratedVectors
- #include <lattice_Zn.h>
recursive sphere codec
Uses a recursive decomposition on the dimensions to encode centroids found by the ZnSphereSearch. The codes are not compatible with the ones of ZnSpehreCodec
Public Functions
-
ZnSphereCodecRec(int dim, int r2)
-
uint64_t encode_centroid(const float *c) const
-
virtual void decode(uint64_t code, float *c) const override
decode it
-
virtual uint64_t encode(const float *x) const override
vectors need to be centroids (does not work on arbitrary vectors)
-
uint64_t get_nv(int ld, int r2a) const
-
uint64_t get_nv_cum(int ld, int r2t, int r2a) const
-
void set_nv_cum(int ld, int r2t, int r2a, uint64_t v)
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
-
ZnSphereCodecRec(int dim, int r2)
-
struct ZnSphereCodecAlt : public faiss::ZnSphereCodec
- #include <lattice_Zn.h>
Codec that uses the recursive codec if dim is a power of 2 and the regular one otherwise
Public Functions
-
ZnSphereCodecAlt(int dim, int r2)
-
virtual uint64_t encode(const float *x) const override
takes vectors that do not need to be centroids
-
virtual void decode(uint64_t code, float *c) const override
decode it
-
uint64_t search_and_encode(const float *x) const
-
float search(const float *x, float *c) const
find nearest centroid. x does not need to be normalized
-
float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const
full call. Requires externally-allocated temp space
-
void search_multi(int n, const float *x, float *c_out, float *dp_out)
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
Public Members
-
bool use_rec
-
ZnSphereCodecRec znc_rec
-
std::vector<CodeSegment> code_segments
-
uint64_t nv
-
size_t code_size
-
int dimS
-
int r2
-
int natom
-
std::vector<float> voc
size dim * ntatom
-
int dim
-
ZnSphereCodecAlt(int dim, int r2)
-
struct LocalSearchQuantizer : public faiss::AdditiveQuantizer
- #include <LocalSearchQuantizer.h>
Implementation of LSQ/LSQ++ described in the following two papers:
Revisiting additive quantization Julieta Martinez, et al. ECCV 2016
LSQ++: Lower running time and higher recall in multi-codebook quantization Julieta Martinez, et al. ECCV 2018
This implementation is mostly translated from the Julia implementations by Julieta Martinez: (https://github.com/una-dinosauria/local-search-quantization, https://github.com/una-dinosauria/Rayuela.jl)
The trained codes are stored in
codebooks
which is calledcentroids
in PQ and RQ.Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
LocalSearchQuantizer(size_t d, size_t M, size_t nbits, Search_type_t search_type = ST_decompress)
-
LocalSearchQuantizer()
-
~LocalSearchQuantizer() override
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
n – number of vectors
centroids – centroids to be added to x, size n * d
-
void update_codebooks(const float *x, const int32_t *codes, size_t n)
Update codebooks given encodings
- Parameters:
x – training vectors, size n * d
codes – encoded training vectors, size n * M
n – number of vectors
-
void icm_encode(int32_t *codes, const float *x, size_t n, size_t ils_iters, std::mt19937 &gen) const
Encode vectors given codebooks using iterative conditional mode (icm).
- Parameters:
codes – output codes, size n * M
x – vectors to encode, size n * d
n – number of vectors
ils_iters – number of iterations of iterative local search
-
void icm_encode_impl(int32_t *codes, const float *x, const float *unaries, std::mt19937 &gen, size_t n, size_t ils_iters, bool verbose) const
-
void icm_encode_step(int32_t *codes, const float *unaries, const float *binaries, size_t n, size_t n_iters) const
-
void perturb_codes(int32_t *codes, size_t n, std::mt19937 &gen) const
Add some perturbation to codes
- Parameters:
codes – codes to be perturbed, size n * M
n – number of vectors
-
void perturb_codebooks(float T, const std::vector<float> &stddev, std::mt19937 &gen)
Add some perturbation to codebooks
- Parameters:
T – temperature of simulated annealing
stddev – standard derivations of each dimension in training data
-
void compute_binary_terms(float *binaries) const
Compute binary terms
- Parameters:
binaries – binary terms, size M * M * K * K
-
void compute_unary_terms(const float *x, float *unaries, size_t n) const
Compute unary terms
- Parameters:
n – number of vectors
x – vectors to encode, size n * d
unaries – unary terms, size n * M * K
-
float evaluate(const int32_t *codes, const float *x, size_t n, float *objs = nullptr) const
Helper function to compute reconstruction error
- Parameters:
codes – encoded codes, size n * M
x – vectors to encode, size n * d
n – number of vectors
objs – if it is not null, store reconstruction error of each vector into it, size n
-
void compute_codebook_tables()
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const
Compute inner-product look-up tables. Used in the centroid search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t K
number of codes per codebook
-
size_t train_iters = 25
number of iterations in training
-
size_t encode_ils_iters = 16
iterations of local search in encoding
-
size_t train_ils_iters = 8
iterations of local search in training
-
size_t icm_iters = 4
number of iterations in icm
-
float p = 0.5f
temperature factor
-
float lambd = 1e-2f
regularization factor
-
size_t chunk_size = 10000
nb of vectors to encode at a time
-
int random_seed = 0x12345
seed for random generator
-
size_t nperts = 4
number of perturbation in each code
if non-NULL, use this encoder to encode (owned by the object)
-
lsq::IcmEncoderFactory *icm_encoder_factory = nullptr
-
bool update_codebooks_with_double = true
-
size_t M
number of codebooks
-
std::vector<uint64_t> codebook_offsets
codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
std::vector<float> norm_tabs
auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan
-
IndexFlat1D qnorm
store and search norms
-
std::vector<float> codebook_cross_products
dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct DummyScaler
- #include <LookupTableScaler.h>
no-op handler
Public Functions
-
inline simd32uint8 lookup(const simd32uint8&, const simd32uint8&) const
-
inline simd16uint16 scale_lo(const simd32uint8&) const
-
inline simd16uint16 scale_hi(const simd32uint8&) const
Public Static Attributes
-
static constexpr int nscale = 0
-
inline simd32uint8 lookup(const simd32uint8&, const simd32uint8&) const
-
struct NormTableScaler
- #include <LookupTableScaler.h>
consumes 2x4 bits to encode a norm as a scalar additive quantizer the norm is scaled because its range if larger than other components
Public Functions
-
inline explicit NormTableScaler(int scale)
-
inline simd32uint8 lookup(const simd32uint8 &lut, const simd32uint8 &c) const
-
inline simd16uint16 scale_lo(const simd32uint8 &res) const
-
inline simd16uint16 scale_hi(const simd32uint8 &res) const
Public Static Attributes
-
static constexpr int nscale = 2
-
inline explicit NormTableScaler(int scale)
-
struct NNDescent
-
Public Functions
-
explicit NNDescent(const int d, const int K)
-
~NNDescent()
-
void build(DistanceComputer &qdis, const int n, bool verbose)
-
void search(DistanceComputer &qdis, const int topk, idx_t *indices, float *dists, VisitedTable &vt) const
-
void reset()
-
void init_graph(DistanceComputer &qdis)
Initialize the KNN graph randomly.
-
void nndescent(DistanceComputer &qdis, bool verbose)
Perform NNDescent algorithm.
-
void join(DistanceComputer &qdis)
Perform local join on each node.
-
void update()
Sample new neighbors for each node to peform local join later.
-
void generate_eval_set(DistanceComputer &qdis, std::vector<int> &c, std::vector<std::vector<int>> &v, int N)
Sample a small number of points to evaluate the quality of KNNG built.
-
explicit NNDescent(const int d, const int K)
-
struct NSG
Public Types
-
using storage_idx_t = int32_t
internal storage of vectors (32 bits: this is expensive)
-
using Node = nsg::Node
-
using Neighbor = nsg::Neighbor
Public Functions
-
explicit NSG(int R = 32)
-
void reset()
-
void search(DistanceComputer &dis, int k, idx_t *I, float *D, VisitedTable &vt) const
-
template<bool collect_fullset, class index_t>
void search_on_graph(const nsg::Graph<index_t> &graph, DistanceComputer &dis, VisitedTable &vt, int ep, int pool_size, std::vector<Neighbor> &retset, std::vector<Node> &fullset) const
-
void add_reverse_links(int q, std::vector<std::mutex> &locks, DistanceComputer &dis, nsg::Graph<Node> &graph)
-
void sync_prune(int q, std::vector<Node> &pool, DistanceComputer &dis, VisitedTable &vt, const nsg::Graph<idx_t> &knn_graph, nsg::Graph<Node> &graph)
-
void link(Index *storage, const nsg::Graph<idx_t> &knn_graph, nsg::Graph<Node> &graph, bool verbose)
-
int dfs(VisitedTable &vt, int root, int cnt) const
-
int attach_unlinked(Index *storage, VisitedTable &vt, VisitedTable &vt2, std::vector<int> °rees)
-
void check_graph() const
Public Members
-
int ntotal = 0
nb of nodes
-
int R
nb of neighbors per node
-
int L
length of the search path at construction time
-
int C
candidate pool size at construction time
-
int search_L = 16
length of the search path
-
int enterpoint
enterpoint
-
bool is_built = false
NSG is built or not.
-
RandomGenerator rng
random generator
-
using storage_idx_t = int32_t
-
struct SimulatedAnnealingParameters
- #include <PolysemousTraining.h>
parameters used for the simulated annealing method
Subclassed by faiss::PolysemousTraining, faiss::SimulatedAnnealingOptimizer
Public Functions
-
inline SimulatedAnnealingParameters()
Public Members
-
double init_temperature = 0.7
-
double temperature_decay = 0.9997893011688015
-
int n_iter = 500000
-
int n_redo = 2
-
int seed = 123
-
int verbose = 0
-
bool only_bit_flips = false
-
bool init_random = false
-
inline SimulatedAnnealingParameters()
-
struct PermutationObjective
- #include <PolysemousTraining.h>
abstract class for the loss function
Subclassed by faiss::ReproduceDistancesObjective
Public Functions
-
virtual double compute_cost(const int *perm) const = 0
-
virtual double cost_update(const int *perm, int iw, int jw) const
-
inline virtual ~PermutationObjective()
Public Members
-
int n
-
virtual double compute_cost(const int *perm) const = 0
-
struct ReproduceDistancesObjective : public faiss::PermutationObjective
Public Functions
-
double dis_weight(double x) const
-
double get_source_dis(int i, int j) const
-
virtual double compute_cost(const int *perm) const override
-
virtual double cost_update(const int *perm, int iw, int jw) const override
-
ReproduceDistancesObjective(int n, const double *source_dis_in, const double *target_dis_in, double dis_weight_factor)
-
void set_affine_target_dis(const double *source_dis_in)
-
inline ~ReproduceDistancesObjective() override
Public Members
-
double dis_weight_factor
-
std::vector<double> source_dis
“real” corrected distances (size n^2)
-
const double *target_dis
wanted distances (size n^2)
-
std::vector<double> weights
weights for each distance (size n^2)
-
int n
Public Static Functions
-
static inline double sqr(double x)
-
static void compute_mean_stdev(const double *tab, size_t n2, double *mean_out, double *stddev_out)
-
double dis_weight(double x) const
-
struct SimulatedAnnealingOptimizer : public faiss::SimulatedAnnealingParameters
- #include <PolysemousTraining.h>
Simulated annealing optimization algorithm for permutations.
Public Functions
-
SimulatedAnnealingOptimizer(PermutationObjective *obj, const SimulatedAnnealingParameters &p)
logs values of the cost function
-
double optimize(int *perm)
-
double run_optimization(int *best_perm)
-
virtual ~SimulatedAnnealingOptimizer()
Public Members
-
PermutationObjective *obj
-
int n
size of the permutation
-
FILE *logfile
-
RandomGenerator *rnd
-
double init_cost
remember initial cost of optimization
-
double init_temperature = 0.7
-
double temperature_decay = 0.9997893011688015
-
int n_iter = 500000
-
int n_redo = 2
-
int seed = 123
-
int verbose = 0
-
bool only_bit_flips = false
-
bool init_random = false
-
SimulatedAnnealingOptimizer(PermutationObjective *obj, const SimulatedAnnealingParameters &p)
-
struct PolysemousTraining : public faiss::SimulatedAnnealingParameters
- #include <PolysemousTraining.h>
optimizes the order of indices in a ProductQuantizer
Public Types
-
enum Optimization_type_t
Values:
-
enumerator OT_None
-
enumerator OT_ReproduceDistances_affine
default
-
enumerator OT_Ranking_weighted_diff
same as _2, but use rank of y+ - rank of y-
-
enumerator OT_None
Public Functions
-
PolysemousTraining()
-
void optimize_pq_for_hamming(ProductQuantizer &pq, size_t n, const float *x) const
reorder the centroids so that the Hamming distance becomes a good approximation of the SDC distance (called by train)
-
void optimize_ranking(ProductQuantizer &pq, size_t n, const float *x) const
called by optimize_pq_for_hamming
-
void optimize_reproduce_distances(ProductQuantizer &pq) const
called by optimize_pq_for_hamming
-
size_t memory_usage_per_thread(const ProductQuantizer &pq) const
make sure we don’t blow up the memory
Public Members
-
Optimization_type_t optimization_type
-
int ntrain_permutation
use 1/4 of the training points for the optimization, with max. ntrain_permutation. If ntrain_permutation == 0: train on centroids
-
double dis_weight_factor
decay of exp that weights distance loss
-
size_t max_memory
refuse to train if it would require more than that amount of RAM
-
std::string log_pattern
-
double init_temperature = 0.7
-
double temperature_decay = 0.9997893011688015
-
int n_iter = 500000
-
int n_redo = 2
-
int seed = 123
-
int verbose = 0
-
bool only_bit_flips = false
-
bool init_random = false
-
enum Optimization_type_t
-
struct CodePackerPQ4 : public faiss::CodePacker
- #include <pq4_fast_scan.h>
CodePacker API for the PQ4 fast-scan
Public Functions
-
CodePackerPQ4(size_t nsq, size_t bbs)
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const final
-
virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const final
-
virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const
-
virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const
-
CodePackerPQ4(size_t nsq, size_t bbs)
-
struct ProductAdditiveQuantizer : public faiss::AdditiveQuantizer
- #include <ProductAdditiveQuantizer.h>
Product Additive Quantizers
The product additive quantizer is a variant of AQ and PQ. It first splits the vector space into multiple orthogonal sub-spaces just like PQ does. And then it quantizes each sub-space by an independent additive quantizer.
Subclassed by faiss::ProductLocalSearchQuantizer, faiss::ProductResidualQuantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ProductAdditiveQuantizer(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type = ST_decompress)
Construct a product additive quantizer.
The additive quantizers passed in will be cloned into the ProductAdditiveQuantizer object.
- Parameters:
d – dimensionality of the input vectors
aqs – sub-additive quantizers
search_type – AQ search type
-
ProductAdditiveQuantizer()
-
virtual ~ProductAdditiveQuantizer()
-
void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)
-
AdditiveQuantizer *subquantizer(size_t m) const
Train the product additive quantizer.
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override
Compute inner-product look-up tables. Used in the search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void compute_codebook_tables()
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t nsplits
number of sub-vectors we split a vector into
-
std::vector<AdditiveQuantizer*> quantizers
-
size_t M
number of codebooks
-
std::vector<uint64_t> codebook_offsets
codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
std::vector<float> norm_tabs
auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan
-
IndexFlat1D qnorm
store and search norms
-
std::vector<float> codebook_cross_products
dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct ProductLocalSearchQuantizer : public faiss::ProductAdditiveQuantizer
- #include <ProductAdditiveQuantizer.h>
Product Local Search Quantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ProductLocalSearchQuantizer(size_t d, size_t nsplits, size_t Msub, size_t nbits, Search_type_t search_type = ST_decompress)
Construct a product LSQ object.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of sub-vectors we split a vector into
Msub – number of codebooks of each LSQ
nbits – bits for each step
search_type – AQ search type
-
ProductLocalSearchQuantizer()
-
void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)
-
AdditiveQuantizer *subquantizer(size_t m) const
Train the product additive quantizer.
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override
Compute inner-product look-up tables. Used in the search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void compute_codebook_tables()
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t nsplits
number of sub-vectors we split a vector into
-
std::vector<AdditiveQuantizer*> quantizers
-
size_t M
number of codebooks
-
std::vector<uint64_t> codebook_offsets
codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
std::vector<float> norm_tabs
auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan
-
IndexFlat1D qnorm
store and search norms
-
std::vector<float> codebook_cross_products
dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct ProductResidualQuantizer : public faiss::ProductAdditiveQuantizer
- #include <ProductAdditiveQuantizer.h>
Product Residual Quantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ProductResidualQuantizer(size_t d, size_t nsplits, size_t Msub, size_t nbits, Search_type_t search_type = ST_decompress)
Construct a product RQ object.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of sub-vectors we split a vector into
Msub – number of codebooks of each RQ
nbits – bits for each step
search_type – AQ search type
-
ProductResidualQuantizer()
-
void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)
-
AdditiveQuantizer *subquantizer(size_t m) const
Train the product additive quantizer.
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override
Compute inner-product look-up tables. Used in the search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void compute_codebook_tables()
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t nsplits
number of sub-vectors we split a vector into
-
std::vector<AdditiveQuantizer*> quantizers
-
size_t M
number of codebooks
-
std::vector<uint64_t> codebook_offsets
codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
std::vector<float> norm_tabs
auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan
-
IndexFlat1D qnorm
store and search norms
-
std::vector<float> codebook_cross_products
dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct ProductQuantizer : public faiss::Quantizer
- #include <ProductQuantizer.h>
Product Quantizer. PQ is trained using k-means, minimizing the L2 distance to centroids. PQ supports L2 and Inner Product search, however the quantization error is biased towards L2 distance.
Public Types
-
enum train_type_t
initialization
Values:
-
enumerator Train_default
-
enumerator Train_hot_start
the centroids are already initialized
share dictionary across PQ segments
-
enumerator Train_hypercube
initialize centroids with nbits-D hypercube
-
enumerator Train_hypercube_pca
initialize centroids with nbits-D hypercube
-
enumerator Train_default
Public Functions
-
inline float *get_centroids(size_t m, size_t i)
return the centroids associated with subvector m
-
inline const float *get_centroids(size_t m, size_t i) const
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
ProductQuantizer(size_t d, size_t M, size_t nbits)
-
ProductQuantizer()
-
void set_derived_values()
compute derived values when d, M and nbits have been set
-
void set_params(const float *centroids, int m)
Define the centroids for subquantizer m.
-
void compute_code(const float *x, uint8_t *code) const
Quantize one vector with the product quantizer.
-
virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
same as compute_code for several vectors
-
void compute_codes_with_assign_index(const float *x, uint8_t *codes, size_t n)
speed up code assignment using assign_index (non-const because the index is changed)
-
void decode(const uint8_t *code, float *x) const
decode a vector from a given code (or n vectors if third argument)
-
virtual void decode(const uint8_t *code, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – input codes, size n * code_size
x – output vectors, size n * d
-
void compute_code_from_distance_table(const float *tab, uint8_t *code) const
If we happen to have the distance tables precomputed, this is more efficient to compute the codes.
-
void compute_distance_table(const float *x, float *dis_table) const
Compute distance table for one vector.
The distance table for x = [x_0 x_1 .. x_(M-1)] is a M * ksub matrix that contains
dis_table (m, j) = || x_m - c_(m, j)||^2 for m = 0..M-1 and j = 0 .. ksub - 1
where c_(m, j) is the centroid no j of sub-quantizer m.
- Parameters:
x – input vector size d
dis_table – output table, size M * ksub
-
void compute_inner_prod_table(const float *x, float *dis_table) const
-
void compute_distance_tables(size_t nx, const float *x, float *dis_tables) const
compute distance table for several vectors
- Parameters:
nx – nb of input vectors
x – input vector size nx * d
dis_table – output table, size nx * M * ksub
-
void compute_inner_prod_tables(size_t nx, const float *x, float *dis_tables) const
-
void search(const float *x, size_t nx, const uint8_t *codes, const size_t ncodes, float_maxheap_array_t *res, bool init_finalize_heap = true) const
perform a search (L2 distance)
- Parameters:
x – query vectors, size nx * d
nx – nb of queries
codes – database codes, size ncodes * code_size
ncodes – nb of nb vectors
res – heap array to store results (nh == nx)
init_finalize_heap – initialize heap (input) and sort (output)?
-
void search_ip(const float *x, size_t nx, const uint8_t *codes, const size_t ncodes, float_minheap_array_t *res, bool init_finalize_heap = true) const
same as search, but with inner product similarity
-
void compute_sdc_table()
-
void search_sdc(const uint8_t *qcodes, size_t nq, const uint8_t *bcodes, const size_t ncodes, float_maxheap_array_t *res, bool init_finalize_heap = true) const
-
void sync_transposed_centroids()
Sync transposed centroids with regular centroids. This call is needed if centroids were edited directly.
-
void clear_transposed_centroids()
Clear transposed centroids table so ones are no longer used.
Public Members
-
size_t M
number of subquantizers
-
size_t nbits
number of bits per quantization index
-
size_t dsub
dimensionality of each subvector
-
size_t ksub
number of centroids for each subquantizer
-
bool verbose
verbose during training?
-
train_type_t train_type
-
ClusteringParameters cp
parameters used during clustering
-
Index *assign_index
if non-NULL, use this index for assignment (should be of size d / M)
-
std::vector<float> centroids
Centroid table, size M * ksub * dsub. Layout: (M, ksub, dsub)
-
std::vector<float> transposed_centroids
Transposed centroid table, size M * ksub * dsub. Layout: (dsub, M, ksub)
-
std::vector<float> centroids_sq_lengths
Squared lengths of centroids, size M * ksub Layout: (M, ksub)
-
std::vector<float> sdc_table
Symmetric Distance Table.
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum train_type_t
-
struct PQEncoderGeneric
Public Functions
-
inline PQEncoderGeneric(uint8_t *code, int nbits, uint8_t offset = 0)
-
inline void encode(uint64_t x)
-
inline ~PQEncoderGeneric()
Public Members
-
uint8_t *code
code for this vector
-
uint8_t offset
-
const int nbits
number of bits per subquantizer index
-
uint8_t reg
-
inline PQEncoderGeneric(uint8_t *code, int nbits, uint8_t offset = 0)
-
struct PQEncoder8
Public Functions
-
inline PQEncoder8(uint8_t *code, int nbits)
-
inline void encode(uint64_t x)
Public Members
-
uint8_t *code
-
inline PQEncoder8(uint8_t *code, int nbits)
-
struct PQEncoder16
Public Functions
-
inline PQEncoder16(uint8_t *code, int nbits)
-
inline void encode(uint64_t x)
Public Members
-
uint16_t *code
-
inline PQEncoder16(uint8_t *code, int nbits)
-
struct PQDecoderGeneric
Public Functions
-
inline PQDecoderGeneric(const uint8_t *code, int nbits)
-
inline uint64_t decode()
Public Members
-
const uint8_t *code
-
uint8_t offset
-
const int nbits
-
const uint64_t mask
-
uint8_t reg
-
inline PQDecoderGeneric(const uint8_t *code, int nbits)
-
struct PQDecoder8
Public Functions
-
inline PQDecoder8(const uint8_t *code, int nbits)
-
inline uint64_t decode()
Public Members
-
const uint8_t *code
Public Static Attributes
-
static const int nbits = 8
-
inline PQDecoder8(const uint8_t *code, int nbits)
-
struct PQDecoder16
Public Functions
-
inline PQDecoder16(const uint8_t *code, int nbits)
-
inline uint64_t decode()
Public Members
-
const uint16_t *code
Public Static Attributes
-
static const int nbits = 16
-
inline PQDecoder16(const uint8_t *code, int nbits)
-
struct Quantizer
- #include <Quantizer.h>
General interface for quantizer objects
Subclassed by faiss::AdditiveQuantizer, faiss::ProductQuantizer, faiss::ScalarQuantizer
Public Functions
-
inline explicit Quantizer(size_t d = 0, size_t code_size = 0)
-
virtual void train(size_t n, const float *x) = 0
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const = 0
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
virtual void decode(const uint8_t *code, float *x, size_t n) const = 0
Decode a set of vectors
- Parameters:
codes – input codes, size n * code_size
x – output vectors, size n * d
-
inline virtual ~Quantizer()
Public Members
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
inline explicit Quantizer(size_t d = 0, size_t code_size = 0)
-
struct ResidualQuantizer : public faiss::AdditiveQuantizer
- #include <ResidualQuantizer.h>
Residual quantizer with variable number of bits per sub-quantizer
The residual centroids are stored in a big cumulative centroid table. The codes are represented either as a non-compact table of size (n, M) or as the compact output (n, code_size).
Public Types
-
using train_type_t = int
initialization
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ResidualQuantizer(size_t d, const std::vector<size_t> &nbits, Search_type_t search_type = ST_decompress)
-
ResidualQuantizer(size_t d, size_t M, size_t nbits, Search_type_t search_type = ST_decompress)
-
ResidualQuantizer()
-
virtual void train(size_t n, const float *x) override
Train the residual quantizer.
-
void initialize_from(const ResidualQuantizer &other, int skip_M = 0)
Copy the M codebook levels from other, starting from skip_M.
-
float retrain_AQ_codebook(size_t n, const float *x)
Encode the vectors and compute codebook that minimizes the quantization error on these codes
- Parameters:
x – training vectors, size n * d
n – nb of training vectors, n >= total_codebook_size
- Returns:
returns quantization error for the new codebook with old codes
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void refine_beam(size_t n, size_t beam_size, const float *residuals, int new_beam_size, int32_t *new_codes, float *new_residuals = nullptr, float *new_distances = nullptr) const
lower-level encode function
- Parameters:
n – number of vectors to handle
residuals – vectors to encode, size (n, beam_size, d)
beam_size – input beam size
new_beam_size – output beam size (should be <= K * beam_size)
new_codes – output codes, size (n, new_beam_size, m + 1)
new_residuals – output residuals, size (n, new_beam_size, d)
new_distances – output distances, size (n, new_beam_size)
-
void refine_beam_LUT(size_t n, const float *query_norms, const float *query_cp, int new_beam_size, int32_t *new_codes, float *new_distances = nullptr) const
-
size_t memory_per_point(int beam_size = -1) const
Beam search can consume a lot of memory. This function estimates the amount of mem used by refine_beam to adjust the batch size
- Parameters:
beam_size – if != -1, override the beam size
-
void compute_codebook_tables()
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const
Compute inner-product look-up tables. Used in the centroid search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
train_type_t train_type = Train_progressive_dim
Binary or of the Train_* flags below.
-
int niter_codebook_refine = 5
number of iterations for codebook refinement.
-
int max_beam_size = 5
beam size used for training and for encoding
-
int use_beam_LUT = 0
use LUT for beam search
-
ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK
Currently used mode of approximate min-k computations. Default value is EXACT_TOPK.
-
ProgressiveDimClusteringParameters cp
clustering parameters
-
ProgressiveDimIndexFactory *assign_index_factory = nullptr
if non-NULL, use this index for assignment
-
size_t M
number of codebooks
-
std::vector<uint64_t> codebook_offsets
codebook #1 is stored in rows codebook_offsets[i]:codebook_offsets[i+1] in the codebooks table of size total_codebook_size by d
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
std::vector<float> norm_tabs
auxiliary data for ST_norm_lsq2x4 and ST_norm_rq2x4 store norms of codebook entries for 4-bit fastscan
-
IndexFlat1D qnorm
store and search norms
-
std::vector<float> codebook_cross_products
dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
Public Static Attributes
-
static const int Train_default = 0
regular k-means (minimal amount of computation)
-
static const int Train_progressive_dim = 1
progressive dim clustering (set by default)
-
static const int Train_refine_codebook = 2
do a few iterations of codebook refinement after first level estimation
-
static const int Train_top_beam = 1024
set this bit on train_type if beam is to be trained only on the first element of the beam (faster but less accurate)
-
static const int Skip_codebook_tables = 2048
set this bit to not autmatically compute the codebook tables after training
-
using train_type_t = int
-
template<class C, bool use_sel = false>
struct BlockResultHandler Public Functions
-
inline explicit BlockResultHandler(size_t nq, const IDSelector *sel = nullptr)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2)
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline virtual void end_multiple()
-
inline virtual ~BlockResultHandler()
-
inline bool is_in_selection(idx_t i) const
-
inline explicit BlockResultHandler(size_t nq, const IDSelector *sel = nullptr)
-
template<class C, bool use_sel = false>
struct Top1BlockResultHandler : public faiss::BlockResultHandler<C, false> -
Public Functions
-
inline Top1BlockResultHandler(size_t nq, T *dis_tab, TI *ids_tab, const IDSelector *sel = nullptr)
-
inline virtual void begin_multiple(size_t i0, size_t i1) final
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab_2) final
add results for query i0..i1 and j0..j1
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline virtual void end_multiple()
-
inline bool is_in_selection(idx_t i) const
-
struct SingleResultHandler : public faiss::ResultHandler<C>
Public Functions
-
inline explicit SingleResultHandler(Top1BlockResultHandler &hr)
-
inline void begin(const size_t current_idx_2)
begin results for query # i
-
inline void end()
series of results for query i is done
-
inline explicit SingleResultHandler(Top1BlockResultHandler &hr)
-
inline Top1BlockResultHandler(size_t nq, T *dis_tab, TI *ids_tab, const IDSelector *sel = nullptr)
-
template<class C, bool use_sel = false>
struct HeapBlockResultHandler : public faiss::BlockResultHandler<C, false> -
Public Functions
-
inline HeapBlockResultHandler(size_t nq, T *heap_dis_tab, TI *heap_ids_tab, size_t k, const IDSelector *sel = nullptr)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2) final
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab) final
add results for query i0..i1 and j0..j1
-
inline virtual void end_multiple() final
series of results for queries i0..i1 is done
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline bool is_in_selection(idx_t i) const
Public Members
-
T *heap_dis_tab
-
TI *heap_ids_tab
-
int64_t k
-
size_t nq
-
const IDSelector *sel
-
size_t i0
-
size_t i1
-
struct SingleResultHandler : public faiss::ResultHandler<C>
Public Functions
-
inline explicit SingleResultHandler(HeapBlockResultHandler &hr)
-
inline void begin(size_t i)
begin results for query # i
-
inline void end()
series of results for query i is done
-
inline explicit SingleResultHandler(HeapBlockResultHandler &hr)
-
inline HeapBlockResultHandler(size_t nq, T *heap_dis_tab, TI *heap_ids_tab, size_t k, const IDSelector *sel = nullptr)
-
template<class C>
struct ReservoirTopN : public faiss::ResultHandler<C> - #include <ResultHandler.h>
Reservoir for a single query.
Subclassed by faiss::ReservoirBlockResultHandler< C, use_sel >::SingleResultHandler
Public Functions
-
inline ReservoirTopN()
-
inline void shrink_fuzzy()
-
inline void shrink()
-
inline ReservoirTopN()
-
template<class C, bool use_sel = false>
struct ReservoirBlockResultHandler : public faiss::BlockResultHandler<C, false> -
Public Functions
-
inline ReservoirBlockResultHandler(size_t nq, T *heap_dis_tab, TI *heap_ids_tab, size_t k, const IDSelector *sel = nullptr)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2)
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab)
add results for query i0..i1 and j0..j1
-
inline virtual void end_multiple() final
series of results for queries i0..i1 is done
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline bool is_in_selection(idx_t i) const
Public Members
-
T *heap_dis_tab
-
TI *heap_ids_tab
-
int64_t k
-
size_t capacity
-
std::vector<ReservoirTopN<C>> reservoirs
-
size_t nq
-
const IDSelector *sel
-
size_t i0
-
size_t i1
-
struct SingleResultHandler : public faiss::ReservoirTopN<C>
-
Public Functions
-
inline explicit SingleResultHandler(ReservoirBlockResultHandler &hr)
-
inline void begin(size_t qno_2)
begin results for query # i
-
inline void end()
series of results for query qno is done
-
inline void shrink_fuzzy()
-
inline void shrink()
-
inline explicit SingleResultHandler(ReservoirBlockResultHandler &hr)
-
inline ReservoirBlockResultHandler(size_t nq, T *heap_dis_tab, TI *heap_ids_tab, size_t k, const IDSelector *sel = nullptr)
-
template<class C, bool use_sel = false>
struct RangeSearchBlockResultHandler : public faiss::BlockResultHandler<C, false> -
Public Functions
-
inline RangeSearchBlockResultHandler(RangeSearchResult *res, float radius, const IDSelector *sel = nullptr)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2)
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab)
add results for query i0..i1 and j0..j1
-
inline ~RangeSearchBlockResultHandler()
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline virtual void end_multiple()
-
inline bool is_in_selection(idx_t i) const
Public Members
-
RangeSearchResult *res
-
T radius
-
std::vector<RangeSearchPartialResult*> partial_results
-
std::vector<size_t> j0s
-
int pr = 0
-
size_t nq
-
const IDSelector *sel
-
size_t i0
-
size_t i1
-
struct SingleResultHandler : public faiss::ResultHandler<C>
Public Functions
-
inline explicit SingleResultHandler(RangeSearchBlockResultHandler &rh)
-
inline void begin(size_t i)
begin results for query # i
-
inline void end()
series of results for query i is done
-
inline ~SingleResultHandler()
Public Members
-
RangeQueryResult *qr = nullptr
-
inline explicit SingleResultHandler(RangeSearchBlockResultHandler &rh)
-
inline RangeSearchBlockResultHandler(RangeSearchResult *res, float radius, const IDSelector *sel = nullptr)
-
struct ScalarQuantizer : public faiss::Quantizer
- #include <ScalarQuantizer.h>
The uniform quantizer has a range [vmin, vmax]. The range can be the same for all dimensions (uniform) or specific per dimension (default).
Public Types
-
enum QuantizerType
Values:
-
enumerator QT_8bit
8 bits per component
-
enumerator QT_4bit
4 bits per component
-
enumerator QT_8bit_uniform
same, shared range for all dimensions
-
enumerator QT_4bit_uniform
-
enumerator QT_fp16
-
enumerator QT_8bit_direct
fast indexing of uint8s
-
enumerator QT_6bit
6 bits per component
-
enumerator QT_bf16
-
enumerator QT_8bit_direct_signed
fast indexing of signed int8s ranging from [-128 to 127]
-
enumerator QT_8bit
-
enum RangeStat
The uniform encoder can estimate the range of representable values of the unform encoder using different statistics. Here rs = rangestat_arg
Values:
-
enumerator RS_minmax
[min - rs*(max-min), max + rs*(max-min)]
-
enumerator RS_meanstd
[mean - std * rs, mean + std * rs]
-
enumerator RS_quantiles
[Q(rs), Q(1-rs)]
-
enumerator RS_optim
alternate optimization of reconstruction error
-
enumerator RS_minmax
Public Functions
-
ScalarQuantizer(size_t d, QuantizerType qtype)
-
ScalarQuantizer()
-
void set_derived_sizes()
updates internal values based on qtype and d
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
-
virtual void decode(const uint8_t *code, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
SQuantizer *select_quantizer() const
-
SQDistanceComputer *get_distance_computer(MetricType metric = METRIC_L2) const
-
InvertedListScanner *select_InvertedListScanner(MetricType mt, const Index *quantizer, bool store_pairs, const IDSelector *sel, bool by_residual = false) const
Public Members
-
QuantizerType qtype = QT_8bit
-
float rangestat_arg = 0
-
size_t bits = 0
bits per scalar code
-
std::vector<float> trained
trained values (including the range)
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
struct SQDistanceComputer : public faiss::FlatCodesDistanceComputer
Public Functions
-
inline SQDistanceComputer()
-
virtual float query_to_code(const uint8_t *code) const = 0
-
inline virtual float distance_to_code(const uint8_t *code) final
compute distance of current query to an encoded vector
-
virtual void set_query(const float *x) = 0
called before computing distances. Pointer x should remain valid while operator () is called
-
inline SQDistanceComputer()
-
struct SQuantizer
Public Functions
-
virtual void encode_vector(const float *x, uint8_t *code) const = 0
-
virtual void decode_vector(const uint8_t *code, float *x) const = 0
-
inline virtual ~SQuantizer()
-
virtual void encode_vector(const float *x, uint8_t *code) const = 0
-
enum QuantizerType
-
struct SIMDResultHandler
Subclassed by faiss::SIMDResultHandlerToFloat, faiss::simd_result_handlers::DummyResultHandler, faiss::simd_result_handlers::FixedStorageHandler< NQ, BB >, faiss::simd_result_handlers::StoreResultHandler
Public Functions
-
virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
virtual void set_block_origin(size_t i0, size_t j0) = 0
set the sub-matrix that is being computed
-
inline virtual ~SIMDResultHandler()
Public Members
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0
-
struct SIMDResultHandlerToFloat : public faiss::SIMDResultHandler
Subclassed by faiss::simd_result_handlers::ResultHandlerCompare< C, false >, faiss::simd_result_handlers::ResultHandlerCompare< C, with_id_map >
Public Functions
-
inline SIMDResultHandlerToFloat(size_t nq, size_t ntotal)
-
inline virtual void begin(const float *norms)
-
inline virtual void end()
-
virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
virtual void set_block_origin(size_t i0, size_t j0) = 0
set the sub-matrix that is being computed
Public Members
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
inline SIMDResultHandlerToFloat(size_t nq, size_t ntotal)
-
template<typename IndexT>
class ThreadedIndex : public IndexT - #include <ThreadedIndex.h>
A holder of indices in a collection of threads The interface to this class itself is not thread safe
Subclassed by faiss::IndexReplicasTemplate< IndexT >, faiss::IndexShardsTemplate< IndexT >
Public Functions
-
explicit ThreadedIndex(bool threaded)
-
explicit ThreadedIndex(int d, bool threaded)
-
~ThreadedIndex() override
-
virtual void addIndex(IndexT *index)
override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.
-
void removeIndex(IndexT *index)
Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.
-
void runOnIndex(std::function<void(int, IndexT*)> f)
Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)
-
void reset() override
faiss::Index API All indices receive the same call
-
inline int count() const
Returns the number of sub-indices.
-
inline IndexT *at(size_t i)
Returns the i-th sub-index.
-
inline const IndexT *at(size_t i) const
Returns the i-th sub-index (const version)
Public Members
-
bool own_indices = false
Whether or not we are responsible for deleting our contained indices.
Protected Functions
-
virtual void onAfterAddIndex(IndexT *index)
Called just after an index is added.
-
virtual void onAfterRemoveIndex(IndexT *index)
Called just after an index is removed.
-
explicit ThreadedIndex(bool threaded)
-
struct SearchParameters
- #include <Index.h>
Parent class for the optional search paramenters.
Sub-classes with additional search parameters should inherit this class. Ownership of the object fields is always to the caller.
Subclassed by faiss::IndexRefineSearchParameters, faiss::SearchParametersHNSW, faiss::SearchParametersIVF, faiss::SearchParametersPQ, faiss::SearchParametersPreTransform, faiss::SearchParametersResidualCoarseQuantizer, faiss::gpu::SearchParametersCagra
Public Functions
-
inline virtual ~SearchParameters()
make sure we can dynamic_cast this
Public Members
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline virtual ~SearchParameters()
-
struct Index
- #include <Index.h>
Abstract structure for an index, supports adding vectors and searching them.
All vectors provided at add or search time are 32-bit float arrays, although the internal representation may vary.
Subclassed by faiss::AdditiveCoarseQuantizer, faiss::IndexFastScan, faiss::IndexFlatCodes, faiss::IndexHNSW, faiss::IndexIVF, faiss::IndexIVFIndependentQuantizer, faiss::IndexNNDescent, faiss::IndexNSG, faiss::IndexPreTransform, faiss::IndexRandom, faiss::IndexRefine, faiss::IndexRowwiseMinMaxBase, faiss::IndexSplitVectors, faiss::MultiIndexQuantizer, faiss::gpu::GpuIndex
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
inline explicit Index(idx_t d = 0, MetricType metric = METRIC_L2)
-
virtual ~Index()
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) = 0
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const = 0
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reset() = 0
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using component_t = float
-
struct Index2Layer : public faiss::IndexFlatCodes
- #include <Index2Layer.h>
Same as an IndexIVFPQ without the inverted lists: codes are stored sequentially
The class is mainly inteded to store encoded vectors that can be accessed randomly, the search function is not implemented.
Public Functions
-
Index2Layer(Index *quantizer, size_t nlist, int M, int nbit = 8, MetricType metric = METRIC_L2)
-
Index2Layer()
-
~Index2Layer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
not implemented
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void transfer_to_IVFPQ(IndexIVFPQ &other) const
transfer the flat codes to an IVFPQ index
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
Level1Quantizer q1
first level quantizer
-
ProductQuantizer pq
second level quantizer is always a PQ
-
size_t code_size_1
size of the code for the first level (ceil(log8(q1.nlist)))
-
size_t code_size_2
size of the code for the second level
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
Index2Layer(Index *quantizer, size_t nlist, int M, int nbit = 8, MetricType metric = METRIC_L2)
-
struct IndexAdditiveQuantizer : public faiss::IndexFlatCodes
- #include <IndexAdditiveQuantizer.h>
Abstract class for additive quantizers. The search functions are in common.
Subclassed by faiss::IndexLocalSearchQuantizer, faiss::IndexProductLocalSearchQuantizer, faiss::IndexProductResidualQuantizer, faiss::IndexResidualQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
explicit IndexAdditiveQuantizer(idx_t d, AdditiveQuantizer *aq, MetricType metric = METRIC_L2)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexResidualQuantizer : public faiss::IndexAdditiveQuantizer
- #include <IndexAdditiveQuantizer.h>
Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexResidualQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexResidualQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
-
IndexResidualQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexLocalSearchQuantizer : public faiss::IndexAdditiveQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexLocalSearchQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexLocalSearchQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductResidualQuantizer : public faiss::IndexAdditiveQuantizer
- #include <IndexAdditiveQuantizer.h>
Index based on a product residual quantizer.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductResidualQuantizer(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
-
IndexProductResidualQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ProductResidualQuantizer prq
The product residual quantizer used to encode the vectors.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductLocalSearchQuantizer : public faiss::IndexAdditiveQuantizer
- #include <IndexAdditiveQuantizer.h>
Index based on a product local search quantizer.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductLocalSearchQuantizer(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
-
IndexProductLocalSearchQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ProductLocalSearchQuantizer plsq
The product local search quantizer used to encode the vectors.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct AdditiveCoarseQuantizer : public faiss::Index
- #include <IndexAdditiveQuantizer.h>
A “virtual” index where the elements are the residual quantizer centroids.
Intended for use as a coarse quantizer in an IndexIVF.
Subclassed by faiss::LocalSearchCoarseQuantizer, faiss::ResidualCoarseQuantizer
Public Functions
-
explicit AdditiveCoarseQuantizer(idx_t d = 0, AdditiveQuantizer *aq = nullptr, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
explicit AdditiveCoarseQuantizer(idx_t d = 0, AdditiveQuantizer *aq = nullptr, MetricType metric = METRIC_L2)
-
struct SearchParametersResidualCoarseQuantizer : public faiss::SearchParameters
Public Functions
-
inline ~SearchParametersResidualCoarseQuantizer()
Public Members
-
float beam_factor = 4.0f
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline ~SearchParametersResidualCoarseQuantizer()
-
struct ResidualCoarseQuantizer : public faiss::AdditiveCoarseQuantizer
- #include <IndexAdditiveQuantizer.h>
The ResidualCoarseQuantizer is a bit specialized compared to the default AdditiveCoarseQuantizer because it can use a beam search at search time (slow but may be useful for very large vocabularies)
Public Functions
-
void set_beam_factor(float new_beam_factor)
computes centroid norms if required
-
ResidualCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
ResidualCoarseQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
void initialize_from(const ResidualCoarseQuantizer &other)
Copy the M first codebook levels from other. Useful to crop a ResidualQuantizer to its first M quantizers.
-
ResidualCoarseQuantizer()
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
float beam_factor = 4.0f
factor between the beam size and the search k if negative, use exact search-to-centroid
-
std::vector<float> centroid_norms
norms of centroids, useful for knn-search
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
void set_beam_factor(float new_beam_factor)
-
struct LocalSearchCoarseQuantizer : public faiss::AdditiveCoarseQuantizer
-
Public Functions
-
LocalSearchCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
LocalSearchCoarseQuantizer()
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
LocalSearchQuantizer lsq
The residual quantizer used to encode the vectors.
-
std::vector<float> centroid_norms
norms of centroids, useful for knn-search
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
LocalSearchCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
-
struct IndexAdditiveQuantizerFastScan : public faiss::IndexFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Fast scan version of IndexAQ. Works for 4-bit AQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Subclassed by faiss::IndexLocalSearchQuantizerFastScan, faiss::IndexProductLocalSearchQuantizerFastScan, faiss::IndexProductResidualQuantizerFastScan, faiss::IndexResidualQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
explicit IndexAdditiveQuantizerFastScan(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
IndexAdditiveQuantizerFastScan()
-
~IndexAdditiveQuantizerFastScan() override
-
explicit IndexAdditiveQuantizerFastScan(const IndexAdditiveQuantizer &orig, int bbs = 32)
build from an existing IndexAQ
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexResidualQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexResidualQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexLocalSearchQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a local search quantizer. Stored vectors are approximated by local search quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexLocalSearchQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductResidualQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a product residual quantizer. Stored vectors are approximated by product residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductResidualQuantizerFastScan(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexProductResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
ProductResidualQuantizer prq
The product residual quantizer used to encode the vectors.
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductLocalSearchQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a product local search quantizer. Stored vectors are approximated by product local search quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductLocalSearchQuantizerFastScan(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexProductLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
ProductLocalSearchQuantizer plsq
The product local search quantizer used to encode the vectors.
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexBinary
- #include <IndexBinary.h>
Abstract structure for a binary index.
Supports adding vertices and searching them.
All queries are symmetric because there is no distinction between codes and vectors.
Subclassed by faiss::IndexBinaryFlat, faiss::IndexBinaryFromFloat, faiss::IndexBinaryHNSW, faiss::IndexBinaryHash, faiss::IndexBinaryIVF, faiss::IndexBinaryMultiHash, faiss::gpu::GpuIndexBinaryFlat
Public Types
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
explicit IndexBinary(idx_t d = 0, MetricType metric = METRIC_L2)
-
virtual ~IndexBinary()
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add(idx_t n, const uint8_t *x) = 0
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const = 0
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual void reset() = 0
Removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Same as add_with_ids for IndexBinary.
Public Members
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
idx_t ntotal = 0
total nb of indexed vectors
-
bool verbose = false
verbosity level
-
bool is_trained = true
set if the Index does not require training, or if training is done already
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
using component_t = uint8_t
-
struct IndexBinaryFlat : public faiss::IndexBinary
- #include <IndexBinaryFlat.h>
Index that stores the full vectors and performs exhaustive search.
Public Functions
-
explicit IndexBinaryFlat(idx_t d)
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual size_t remove_ids(const IDSelector &sel) override
Remove some ids. Note that because of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted.
-
inline IndexBinaryFlat()
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Same as add_with_ids for IndexBinary.
Public Members
-
std::vector<uint8_t> xb
database vectors, size ntotal * d / 8
-
bool use_heap = true
Select between using a heap or counting to select the k smallest values when scanning inverted lists.
-
size_t query_batch_size = 32
-
ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
explicit IndexBinaryFlat(idx_t d)
-
struct IndexBinaryFromFloat : public faiss::IndexBinary
- #include <IndexBinaryFromFloat.h>
IndexBinary backed by a float Index.
Supports adding vertices and searching them.
All queries are symmetric because there is no distinction between codes and vectors.
Public Functions
-
IndexBinaryFromFloat()
-
explicit IndexBinaryFromFloat(Index *index)
-
~IndexBinaryFromFloat()
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void train(idx_t n, const uint8_t *x) override
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Same as add_with_ids for IndexBinary.
Public Members
-
Index *index = nullptr
-
bool own_fields = false
Whether object owns the index pointer.
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
IndexBinaryFromFloat()
-
struct IndexBinaryHash : public faiss::IndexBinary
- #include <IndexBinaryHash.h>
just uses the b first bits as a hash value
Public Types
-
using InvertedListMap = std::unordered_map<idx_t, InvertedList>
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
IndexBinaryHash(int d, int b)
-
IndexBinaryHash()
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
void display() const
-
size_t hashtable_size() const
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Same as add_with_ids for IndexBinary.
Public Members
-
InvertedListMap invlists
-
int b
-
int nflip
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
struct InvertedList
Public Functions
-
void add(idx_t id, size_t code_size, const uint8_t *code)
-
void add(idx_t id, size_t code_size, const uint8_t *code)
-
using InvertedListMap = std::unordered_map<idx_t, InvertedList>
-
struct IndexBinaryHashStats
Public Functions
-
inline IndexBinaryHashStats()
-
void reset()
Public Members
-
size_t nq
-
size_t n0
-
size_t nlist
-
size_t ndis
-
inline IndexBinaryHashStats()
-
struct IndexBinaryMultiHash : public faiss::IndexBinary
- #include <IndexBinaryHash.h>
just uses the b first bits as a hash value
Public Functions
-
IndexBinaryMultiHash(int d, int nhash, int b)
-
IndexBinaryMultiHash()
-
~IndexBinaryMultiHash()
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
size_t hashtable_size() const
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Same as add_with_ids for IndexBinary.
Public Members
-
IndexBinaryFlat *storage
-
bool own_fields
-
int nhash
nb of hash maps
-
int b
nb bits per hash map
-
int nflip
nb bit flips to use at search time
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
IndexBinaryMultiHash(int d, int nhash, int b)
-
struct IndexBinaryHNSW : public faiss::IndexBinary
- #include <IndexBinaryHNSW.h>
The HNSW index is a normal random-access index with a HNSW link structure built on top
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
explicit IndexBinaryHNSW()
-
explicit IndexBinaryHNSW(int d, int M = 32)
-
explicit IndexBinaryHNSW(IndexBinary *storage, int M = 32)
-
~IndexBinaryHNSW() override
-
DistanceComputer *get_distance_computer() const
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void train(idx_t n, const uint8_t *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Same as add_with_ids for IndexBinary.
Public Members
-
HNSW hnsw
-
bool own_fields
-
IndexBinary *storage
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexBinaryIVF : public faiss::IndexBinary
- #include <IndexBinaryIVF.h>
Index based on a inverted file (IVF)
In the inverted file, the quantizer (an IndexBinary instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.
Otherwise the object is similar to the IndexIVF
Public Functions
-
IndexBinaryIVF(IndexBinary *quantizer, size_t d, size_t nlist)
The Inverted file takes a quantizer (an IndexBinary) on input, which implements the function mapping a vector to a list identifier. The pointer is borrowed: the quantizer should not be deleted while the IndexBinaryIVF is in use.
-
IndexBinaryIVF()
-
~IndexBinaryIVF() override
-
virtual void reset() override
Removes all elements from the database.
-
virtual void train(idx_t n, const uint8_t *x) override
Trains the quantizer.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void add_core(idx_t n, const uint8_t *x, const idx_t *xids, const idx_t *precomputed_idx)
Implementation of vector addition where the vector assignments are predefined.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void search_preassigned(idx_t n, const uint8_t *x, idx_t k, const idx_t *assign, const int32_t *centroid_dis, int32_t *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr) const
Search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. search() calls this.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
-
virtual BinaryInvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void range_search_preassigned(idx_t n, const uint8_t *x, int radius, const idx_t *assign, const int32_t *centroid_dis, RangeSearchResult *result) const
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d / 8
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairs
flag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d / 8)
-
virtual void reconstruct_from_offset(idx_t list_no, idx_t offset, uint8_t *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairs
set.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void merge_from(IndexBinary &other, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
inline size_t get_list_size(size_t list_no) const
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
void display() const
Display the actual class name and some more info.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Same as add_with_ids for IndexBinary.
Public Members
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = true
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
bool use_heap = true
Select between using a heap or counting to select the k smallest values when scanning inverted lists.
-
bool per_invlist_search = false
collect computations per batch
-
DirectMap direct_map
map for direct access to the elements. Enables reconstruct().
-
IndexBinary *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of possible key values
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
IndexBinaryIVF(IndexBinary *quantizer, size_t d, size_t nlist)
-
struct BinaryInvertedListScanner
Public Functions
-
virtual void set_query(const uint8_t *query_vector) = 0
from now on we handle this query.
-
virtual void set_list(idx_t list_no, uint8_t coarse_dis) = 0
following codes come from this inverted list
-
virtual uint32_t distance_to_code(const uint8_t *code) const = 0
compute a single query-to-code distance
-
virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, int32_t *distances, idx_t *labels, size_t k) const = 0
compute the distances to codes. (distances, labels) should be organized as a min- or max-heap
- Parameters:
n – number of codes to scan
codes – codes to scan (n * code_size)
ids – corresponding ids (ignored if store_pairs)
distances – heap distances (size k)
labels – heap labels (size k)
k – heap size
-
virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, int radius, RangeQueryResult &result) const = 0
-
inline virtual ~BinaryInvertedListScanner()
-
virtual void set_query(const uint8_t *query_vector) = 0
-
struct IndexFastScan : public faiss::Index
- #include <IndexFastScan.h>
Fast scan version of IndexPQ and IndexAQ. Works for 4-bit PQ and AQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions. The trailing codes (padding codes that are added to complete the last code) are garbage.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Subclassed by faiss::IndexAdditiveQuantizerFastScan, faiss::IndexPQFastScan
Public Functions
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
IndexFastScan()
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const = 0
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const = 0
-
void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
struct FastScanStats
Public Functions
-
inline FastScanStats()
-
inline void reset()
Public Members
-
uint64_t t0
-
uint64_t t1
-
uint64_t t2
-
uint64_t t3
-
inline FastScanStats()
-
struct IndexFlat : public faiss::IndexFlatCodes
- #include <IndexFlat.h>
Index that stores the full vectors and performs exhaustive search
Subclassed by faiss::IndexFlatIP, faiss::IndexFlatL2
Public Functions
-
explicit IndexFlat(idx_t d, MetricType metric = METRIC_L2)
- Parameters:
d – dimensionality of the input vectors
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
inline IndexFlat()
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
explicit IndexFlat(idx_t d, MetricType metric = METRIC_L2)
-
struct IndexFlatIP : public faiss::IndexFlat
-
Public Functions
-
inline explicit IndexFlatIP(idx_t d)
-
inline IndexFlatIP()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
inline explicit IndexFlatIP(idx_t d)
-
struct IndexFlatL2 : public faiss::IndexFlat
Subclassed by faiss::IndexFlat1D
Public Functions
-
inline explicit IndexFlatL2(idx_t d)
- Parameters:
d – dimensionality of the input vectors
-
inline IndexFlatL2()
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
void sync_l2norms()
-
void clear_l2norms()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
inline explicit IndexFlatL2(idx_t d)
-
struct IndexFlat1D : public faiss::IndexFlatL2
- #include <IndexFlat.h>
optimized version for 1D “vectors”.
Public Functions
-
explicit IndexFlat1D(bool continuous_update = true)
-
void update_permutation()
if not continuous_update, call this between the last add and the first search
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Warn: the distances returned are L1 not L2.
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
void sync_l2norms()
-
void clear_l2norms()
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
bool continuous_update = true
is the permutation updated continuously?
-
std::vector<float> cached_l2norms
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
explicit IndexFlat1D(bool continuous_update = true)
-
struct IndexFlatCodes : public faiss::Index
- #include <IndexFlatCodes.h>
Index that encodes all vectors as fixed-size codes (size code_size). Storage is in the codes vector
Subclassed by faiss::Index2Layer, faiss::IndexAdditiveQuantizer, faiss::IndexFlat, faiss::IndexLSH, faiss::IndexLattice, faiss::IndexNeuralNetCodec, faiss::IndexPQ, faiss::IndexScalarQuantizer
Public Functions
-
IndexFlatCodes()
-
IndexFlatCodes(size_t code_size, idx_t d, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
default add uses sa_encode
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const
a FlatCodesDistanceComputer offers a distance_to_code method
The default implementation explicitly decodes the vector with sa_decode.
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Search implemented by decoding
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_sa_codes(idx_t n, const uint8_t *x, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
void permute_entries(const idx_t *perm)
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
IndexFlatCodes()
-
struct IndexHNSW : public faiss::Index
- #include <IndexHNSW.h>
The HNSW index is a normal random-access index with a HNSW link structure built on top
Subclassed by faiss::IndexHNSW2Level, faiss::IndexHNSWCagra, faiss::IndexHNSWFlat, faiss::IndexHNSWPQ, faiss::IndexHNSWSQ
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
explicit IndexHNSW(int d = 0, int M = 32, MetricType metric = METRIC_L2)
-
explicit IndexHNSW(Index *storage, int M = 32)
-
~IndexHNSW() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWFlat : public faiss::IndexHNSW
- #include <IndexHNSW.h>
Flat index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSWFlat()
-
IndexHNSWFlat(int d, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWPQ : public faiss::IndexHNSW
- #include <IndexHNSW.h>
PQ index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSWPQ()
-
IndexHNSWPQ(int d, int pq_m, int M, int pq_nbits = 8, MetricType metric = METRIC_L2)
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWSQ : public faiss::IndexHNSW
- #include <IndexHNSW.h>
SQ index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSWSQ()
-
IndexHNSWSQ(int d, ScalarQuantizer::QuantizerType qtype, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSW2Level : public faiss::IndexHNSW
- #include <IndexHNSW.h>
2-level code structure with fast random access
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSW2Level()
-
IndexHNSW2Level(Index *quantizer, size_t nlist, int m_pq, int M)
-
void flip_to_ivf()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWCagra : public faiss::IndexHNSW
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSWCagra()
-
IndexHNSWCagra(int d, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1, const SearchParameters *params = nullptr) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
Public Members
-
bool base_level_only = false
When set to true, the index is immutable. This option is used to copy the knn graph from GpuIndexCagra to the base level of IndexHNSWCagra without adding upper levels. Doing so enables to search the HNSW index, but removes the ability to add vectors.
-
int num_base_level_search_entrypoints = 32
When
base_level_only
is set toTrue
, the search function searches only the base level knn graph of the HNSW index. This parameter selects the entry point by randomly selecting some points and using the best one.
-
HNSW hnsw
-
bool own_fields = false
-
Index *storage = nullptr
-
bool init_level0 = true
-
bool keep_max_size_level0 = false
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
typedef HNSW::storage_idx_t storage_idx_t
-
template<typename IndexT>
struct IndexIDMapTemplate : public IndexT - #include <IndexIDMap.h>
Index that translates search results to ids
Subclassed by faiss::IndexIDMap2Template< IndexT >
Public Types
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexIDMapTemplate(IndexT *index)
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void add(idx_t n, const component_t *x) override
this will fail. Use add_with_ids
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
-
void train(idx_t n, const component_t *x) override
-
void reset() override
-
size_t remove_ids(const IDSelector &sel) override
remove ids adapted to IndexFlat
-
void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
-
void check_compatible_for_merge(const IndexT &otherIndex) const override
-
size_t sa_code_size() const override
-
~IndexIDMapTemplate() override
-
inline IndexIDMapTemplate()
-
using component_t = typename IndexT::component_t
-
template<typename IndexT>
struct IndexIDMap2Template : public faiss::IndexIDMapTemplate<IndexT> - #include <IndexIDMap.h>
same as IndexIDMap but also provides an efficient reconstruction implementation via a 2-way index
Public Types
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexIDMap2Template(IndexT *index)
-
void construct_rev_map()
make the rev_map from scratch
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
-
size_t remove_ids(const IDSelector &sel) override
-
void reconstruct(idx_t key, component_t *recons) const override
-
void check_consistency() const
check that the rev_map and the id_map are in sync
-
inline ~IndexIDMap2Template() override
-
inline IndexIDMap2Template()
-
void add(idx_t n, const component_t *x) override
this will fail. Use add_with_ids
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
-
void train(idx_t n, const component_t *x) override
-
void reset() override
-
void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
-
void check_compatible_for_merge(const IndexT &otherIndex) const override
-
size_t sa_code_size() const override
-
using component_t = typename IndexT::component_t
-
struct IDSelectorTranslated : public faiss::IDSelector
Public Functions
-
inline IDSelectorTranslated(const std::vector<int64_t> &id_map, const IDSelector *sel)
-
inline IDSelectorTranslated(IndexBinaryIDMap &index_idmap, const IDSelector *sel)
-
inline IDSelectorTranslated(IndexIDMap &index_idmap, const IDSelector *sel)
-
inline virtual bool is_member(idx_t id) const override
-
inline IDSelectorTranslated(const std::vector<int64_t> &id_map, const IDSelector *sel)
-
struct Level1Quantizer
- #include <IndexIVF.h>
Encapsulates a quantizer object for the IndexIVF
The class isolates the fields that are independent of the storage of the lists (especially training)
Subclassed by faiss::IndexIVFInterface, faiss::IndexShardsIVF
Public Functions
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
-
Level1Quantizer(Index *quantizer, size_t nlist)
-
Level1Quantizer()
-
~Level1Quantizer()
Public Members
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
-
struct SearchParametersIVF : public faiss::SearchParameters
Subclassed by faiss::IVFPQSearchParameters
Public Functions
-
inline virtual ~SearchParametersIVF()
Public Members
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
SearchParameters *quantizer_params = nullptr
-
void *inverted_list_context = nullptr
context object to pass to InvertedLists
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline virtual ~SearchParametersIVF()
-
struct IndexIVFInterface : public faiss::Level1Quantizer
Subclassed by faiss::IndexIVF, faiss::gpu::GpuIndexIVF
Public Functions
-
inline explicit IndexIVFInterface(Index *quantizer = nullptr, size_t nlist = 0)
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const = 0
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const = 0
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
inline virtual ~IndexIVFInterface()
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
inline explicit IndexIVFInterface(Index *quantizer = nullptr, size_t nlist = 0)
-
struct IndexIVF : public faiss::Index, public faiss::IndexIVFInterface
- #include <IndexIVF.h>
Index based on a inverted file (IVF)
In the inverted file, the quantizer (an Index instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.
The inverted list object is required only after trainng. If none is set externally, an ArrayInvertedLists is used automatically.
At search time, the vector to be searched is also quantized, and only the list corresponding to the quantization index is searched. This speeds up the search by making it non-exhaustive. This can be relaxed using multi-probe search: a few (nprobe) quantization indices are selected and several inverted lists are visited.
Sub-classes implement a post-filtering of the index that refines the distance estimation from the query to databse vectors.
Subclassed by faiss::IndexIVFAdditiveQuantizer, faiss::IndexIVFFastScan, faiss::IndexIVFFlat, faiss::IndexIVFPQ, faiss::IndexIVFScalarQuantizer, faiss::IndexIVFSpectralHash
Public Functions
-
IndexIVF(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
The Inverted file takes a quantizer (an Index) on input, which implements the function mapping a vector to a list identifier.
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add(idx_t n, const float *x) override
Calls add_with_ids with NULL ids.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const = 0
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign)
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairs
flag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairs
set.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
~IndexIVF() override
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
IndexIVF()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
IndexIVF(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
-
struct InvertedListScanner
- #include <IndexIVF.h>
Object that handles a query. The inverted lists to scan are provided externally. The object has a lot of state, but distance_to_code and scan_codes can be called in multiple threads
Public Functions
-
inline InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr)
-
virtual void set_query(const float *query_vector) = 0
from now on we handle this query.
-
virtual void set_list(idx_t list_no, float coarse_dis) = 0
following codes come from this inverted list
-
virtual float distance_to_code(const uint8_t *code) const = 0
compute a single query-to-code distance
-
virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, float *distances, idx_t *labels, size_t k) const
scan a set of codes, compute distances to current query and update heap of results if necessary. Default implementation calls distance_to_code.
- Parameters:
n – number of codes to scan
codes – codes to scan (n * code_size)
ids – corresponding ids (ignored if store_pairs)
distances – heap distances (size k)
labels – heap labels (size k)
k – heap size
- Returns:
number of heap updates performed
-
virtual size_t iterate_codes(InvertedListsIterator *iterator, float *distances, idx_t *labels, size_t k, size_t &list_size) const
-
virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, float radius, RangeQueryResult &result) const
scan a set of codes, compute distances to current query and update results if distances are below radius
(default implementation fails)
-
virtual void iterate_codes_range(InvertedListsIterator *iterator, float radius, RangeQueryResult &result, size_t &list_size) const
-
inline virtual ~InvertedListScanner()
Public Members
-
idx_t list_no = -1
remember current list
-
bool keep_max = false
keep maximum instead of minimum
-
bool store_pairs
store positions in invlists rather than labels
-
const IDSelector *sel
search in this subset of ids
-
size_t code_size = 0
used in default implementation of scan_codes
-
inline InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr)
-
struct IndexIVFStats
-
Public Members
-
size_t nq
-
size_t nlist
-
size_t ndis
-
size_t nheap_updates
-
double quantization_time
-
double search_time
-
size_t nq
-
struct IndexIVFAdditiveQuantizer : public faiss::IndexIVF
- #include <IndexIVFAdditiveQuantizer.h>
Abstract class for IVF additive quantizers. The search functions are in common.
Subclassed by faiss::IndexIVFLocalSearchQuantizer, faiss::IndexIVFProductLocalSearchQuantizer, faiss::IndexIVFProductResidualQuantizer, faiss::IndexIVFResidualQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq, Index *quantizer, size_t d, size_t nlist, MetricType metric = METRIC_L2)
-
explicit IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairs
set.
-
~IndexIVFAdditiveQuantizer() override
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairs
flag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
int use_precomputed_table = 0
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFResidualQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
-
IndexIVFResidualQuantizer()
-
virtual ~IndexIVFResidualQuantizer()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairs
set.
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairs
flag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
int use_precomputed_table = 0
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFLocalSearchQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFLocalSearchQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexIVFLocalSearchQuantizer()
-
virtual ~IndexIVFLocalSearchQuantizer()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairs
set.
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids) override
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairs
flag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
LocalSearchQuantizer lsq
The LSQ quantizer used to encode the vectors.
- int use_precomputed_table =
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using IndexIDMap = IndexIDMapTemplate<Index>