Namespace faiss
-
namespace faiss
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Implementation of k-means clustering with many variants.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. IDSelector is intended to define a subset of vectors to handle (for removal or as subset to search)
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. PQ4 SIMD packing and accumulation functions
The basic kernel accumulates nq query vectors with bbs = nb * 2 * 16 vectors and produces an output matrix for that. It is interesting for nq * nb <= 4, otherwise register spilling becomes too large.
The implementation of these functions is spread over 3 cpp files to reduce parallel compile times. Templates are instantiated explicitly.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. This file contains callbacks for kernels that compute distances.
Throughout the library, vectors are provided as float * pointers. Most algorithms can be optimized when several vectors are processed (added/searched) together in a batch. In this case, they are passed in as a matrix. When n vectors of size d are provided as float * x, component j of vector i is
x[ i * d + j ]
where 0 <= i < n and 0 <= j < d. In other words, matrices are always compact. When specifying the size of the matrix, we call it an n*d matrix, which implies a row-major storage.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. I/O functions can read/write to a filename, a file handle or to an object that abstracts the medium.
The read functions return objects that should be deallocated with delete. All references within these objectes are owned by the object.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Definition of inverted lists + a few common classes that implement the interface.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Since IVF (inverted file) indexes are of so much use for large-scale use cases, we group a few functions related to them in this small library. Most functions work both on IndexIVFs and IndexIVFs embedded within an IndexPreTransform.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. In this file are the implementations of extra metrics beyond L2 and inner product
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Defines a few objects that apply transformations to a set of vectors Often these are pre-processing steps.
Typedefs
-
using IndexIDMap = IndexIDMapTemplate<Index>
-
using IndexBinaryIDMap = IndexIDMapTemplate<IndexBinary>
-
using IndexIDMap2 = IndexIDMap2Template<Index>
-
using IndexBinaryIDMap2 = IndexIDMap2Template<IndexBinary>
-
using IVFSearchParameters = SearchParametersIVF
-
using IndexReplicas = IndexReplicasTemplate<Index>
-
using IndexBinaryReplicas = IndexReplicasTemplate<IndexBinary>
-
using IndexShards = IndexShardsTemplate<Index>
-
using IndexBinaryShards = IndexShardsTemplate<IndexBinary>
-
using ConcatenatedInvertedLists = HStackInvertedLists
-
using idx_t = int64_t
all vector indices are this type
Enums
-
enum MetricType
The metric space for vector comparison for Faiss indices and algorithms.
Most algorithms support both inner product and L2, with the flat (brute-force) indices supporting additional metric types for vector comparison.
Values:
-
enumerator METRIC_INNER_PRODUCT
maximum inner product search
-
enumerator METRIC_L2
squared L2 search
-
enumerator METRIC_L1
L1 (aka cityblock)
-
enumerator METRIC_Linf
infinity distance
-
enumerator METRIC_Lp
L_p distance, p is given by a faiss::Index metric_arg
-
enumerator METRIC_Canberra
some additional metrics defined in scipy.spatial.distance
-
enumerator METRIC_BrayCurtis
-
enumerator METRIC_JensenShannon
-
enumerator METRIC_Jaccard
defined as: sum_i(min(a_i, b_i)) / sum_i(max(a_i, b_i)) where a_i, b_i > 0
-
enumerator METRIC_INNER_PRODUCT
Functions
-
IndexBinary *clone_binary_index(const IndexBinary *index)
-
float kmeans_clustering(size_t d, size_t n, size_t k, const float *x, float *centroids)
simplified interface
- Parameters:
d – dimension of the data
n – nb of training vectors
k – nb of output centroids
x – training set (size n * d)
centroids – output centroids (size k * d)
- Returns:
final quantization error
-
template<typename PQDecoderT>
inline float distance_single_code_generic(const size_t M, const size_t nbits, const float *sim_table, const uint8_t *code) Returns the distance to a single code.
- template<typename PQDecoderT> inline void distance_four_codes_generic (const size_t M, const size_t nbits, const float *sim_table, const uint8_t *__restrict code0, const uint8_t *__restrict code1, const uint8_t *__restrict code2, const uint8_t *__restrict code3, float &result0, float &result1, float &result2, float &result3)
Combines 4 operations of distance_single_code() General-purpose version.
-
template<typename PQDecoderT>
inline float distance_single_code(const size_t M, const size_t nbits, const float *sim_table, const uint8_t *code)
- template<typename PQDecoderT> inline void distance_four_codes (const size_t M, const size_t nbits, const float *sim_table, const uint8_t *__restrict code0, const uint8_t *__restrict code1, const uint8_t *__restrict code2, const uint8_t *__restrict code3, float &result0, float &result1, float &result2, float &result3)
-
void handleExceptions(std::vector<std::pair<int, std::exception_ptr>> &exceptions)
Handle multiple exceptions from worker threads, throwing an appropriate exception that aggregates the information The pair int is the thread that generated the exception
-
std::string demangle_cpp_symbol(const char *name)
make typeids more readable
-
uint32_t fourcc(const char sx[4])
cast a 4-character string to a uint32_t that can be written and read easily
-
uint32_t fourcc(const std::string &sx)
-
void fourcc_inv(uint32_t x, char str[5])
-
std::string fourcc_inv(uint32_t x)
-
std::string fourcc_inv_printable(uint32_t x)
-
void smawk(const idx_t nrows, const idx_t ncols, const float *x, idx_t *argmins)
SMAWK algorithm. Find the row minima of a monotone matrix.
Expose this for testing.
- Parameters:
nrows – number of rows
ncols – number of columns
x – input matrix, size (nrows, ncols)
argmins – argmin of each row
-
double kmeans1d(const float *x, size_t n, size_t nclusters, float *centroids)
Exact 1D K-Means by dynamic programming
From “Fast Exact k-Means, k-Medians and Bregman Divergence Clustering in 1D” Allan Grønlund, Kasper Green Larsen, Alexander Mathiasen, Jesper Sindahl Nielsen, Stefan Schneider, Mingzhou Song, ArXiV’17
Section 2.2
https://arxiv.org/abs/1701.07204
- Parameters:
x – input 1D array
n – input array length
nclusters – number of clusters
centroids – output centroids, size nclusters
- Returns:
imbalancce factor
-
void pq4_pack_codes(const uint8_t *codes, size_t ntotal, size_t M, size_t nb, size_t bbs, size_t nsq, uint8_t *blocks)
Pack codes for consumption by the SIMD kernels. The unused bytes are set to 0.
- Parameters:
codes – input codes, size (ntotal, ceil(M / 2))
ntotal – number of input codes
nb – output number of codes (ntotal rounded up to a multiple of bbs)
nsq – number of sub-quantizers (=M rounded up to a muliple of 2)
bbs – size of database blocks (multiple of 32)
blocks – output array, size nb * nsq / 2.
-
void pq4_pack_codes_range(const uint8_t *codes, size_t M, size_t i0, size_t i1, size_t bbs, size_t nsq, uint8_t *blocks)
Same as pack_codes but write in a given range of the output, leaving the rest untouched. Assumes allocated entries are 0 on input.
- Parameters:
codes – input codes, size (i1 - i0, ceil(M / 2))
i0 – first output code to write
i1 – last output code to write
blocks – output array, size at least ceil(i1 / bbs) * bbs * nsq / 2
-
uint8_t pq4_get_packed_element(const uint8_t *data, size_t bbs, size_t nsq, size_t vector_id, size_t sq)
get a single element from a packed codes table
- Parameters:
vector_id – vector id
sq – subquantizer (< nsq)
-
void pq4_set_packed_element(uint8_t *data, uint8_t code, size_t bbs, size_t nsq, size_t vector_id, size_t sq)
set a single element “code” into a packed codes table
- Parameters:
vector_id – vector id
sq – subquantizer (< nsq)
-
void pq4_pack_LUT(int nq, int nsq, const uint8_t *src, uint8_t *dest)
Pack Look-up table for consumption by the kernel.
- Parameters:
nq – number of queries
nsq – number of sub-quantizers (muliple of 2)
src – input array, size (nq, 16)
dest – output array, size (nq, 16)
-
void pq4_accumulate_loop(int nq, size_t nb, int bbs, int nsq, const uint8_t *codes, const uint8_t *LUT, SIMDResultHandler &res, const NormTableScaler *scaler)
Loop over database elements and accumulate results into result handler
- Parameters:
nq – number of queries
nb – number of database elements
bbs – size of database blocks (multiple of 32)
nsq – number of sub-quantizers (muliple of 2)
codes – packed codes array
LUT – packed look-up table
scaler – scaler to scale the encoded norm
-
int pq4_qbs_to_nq(int qbs)
-
int pq4_preferred_qbs(int nq)
return the preferred decomposition in blocks for a nb of queries.
-
int pq4_pack_LUT_qbs(int fqbs, int nsq, const uint8_t *src, uint8_t *dest)
Pack Look-up table for consumption by the kernel.
- Parameters:
qbs – 4-bit encoded number of query blocks, the total number of queries handled (nq) is deduced from it
nsq – number of sub-quantizers (muliple of 2)
src – input array, size (nq, 16)
dest – output array, size (nq, 16)
- Returns:
nq
-
int pq4_pack_LUT_qbs_q_map(int qbs, int nsq, const uint8_t *src, const int *q_map, uint8_t *dest)
Same as pq4_pack_LUT_qbs, except the source vectors are remapped with q_map
-
void pq4_accumulate_loop_qbs(int qbs, size_t nb, int nsq, const uint8_t *codes, const uint8_t *LUT, SIMDResultHandler &res, const NormTableScaler *scaler = nullptr)
Run accumulation loop.
- Parameters:
qbs – 4-bit encoded number of queries
nb – number of database codes (mutliple of bbs)
nsq – number of sub-quantizers
codes – encoded database vectors (packed)
LUT – look-up table (packed)
res – call-back for the resutls
scaler – scaler to scale the encoded norm
-
void beam_search_encode_step(size_t d, size_t K, const float *cent, size_t n, size_t beam_size, const float *residuals, size_t m, const int32_t *codes, size_t new_beam_size, int32_t *new_codes, float *new_residuals, float *new_distances, Index *assign_index = nullptr, ApproxTopK_mode_t approx_topk = ApproxTopK_mode_t::EXACT_TOPK)
Encode a residual by sampling from a centroid table.
This is a single encoding step the residual quantizer. It allows low-level access to the encoding function, exposed mainly for unit tests.
- Parameters:
n – number of vectors to hanlde
residuals – vectors to encode, size (n, beam_size, d)
cent – centroids, size (K, d)
beam_size – input beam size
m – size of the codes for the previous encoding steps
codes – code array for the previous steps of the beam (n, beam_size, m)
new_beam_size – output beam size (should be <= K * beam_size)
new_codes – output codes, size (n, new_beam_size, m + 1)
new_residuals – output residuals, size (n, new_beam_size, d)
new_distances – output distances, size (n, new_beam_size)
assign_index – if non-NULL, will be used to perform assignment
-
void beam_search_encode_step_tab(size_t K, size_t n, size_t beam_size, const float *codebook_cross_norms, size_t ldc, const uint64_t *codebook_offsets, const float *query_cp, size_t ldqc, const float *cent_norms_i, size_t m, const int32_t *codes, const float *distances, size_t new_beam_size, int32_t *new_codes, float *new_distances, ApproxTopK_mode_t approx_topk = ApproxTopK_mode_t::EXACT_TOPK)
Encode a set of vectors using their dot products with the codebooks
- Parameters:
K – number of vectors in the codebook
n – nb of vectors to encode
beam_size – input beam size
codebook_cross_norms – inner product of this codebook with the m previously encoded codebooks
codebook_offsets – offsets into codebook_cross_norms for each previous codebook
query_cp – dot products of query vectors with ???
cent_norms_i – norms of centroids
-
Index *index_factory(int d, const char *description, MetricType metric = METRIC_L2)
Build and index with the sequence of processing steps described in the string.
-
IndexBinary *index_binary_factory(int d, const char *description)
-
void write_index(const Index *idx, const char *fname)
-
void write_index(const Index *idx, FILE *f)
-
void write_index_binary(const IndexBinary *idx, const char *fname)
-
void write_index_binary(const IndexBinary *idx, FILE *f)
-
void write_index_binary(const IndexBinary *idx, IOWriter *writer)
-
Index *read_index(const char *fname, int io_flags = 0)
-
Index *read_index(FILE *f, int io_flags = 0)
-
IndexBinary *read_index_binary(const char *fname, int io_flags = 0)
-
IndexBinary *read_index_binary(FILE *f, int io_flags = 0)
-
IndexBinary *read_index_binary(IOReader *reader, int io_flags = 0)
-
void write_VectorTransform(const VectorTransform *vt, const char *fname)
-
void write_VectorTransform(const VectorTransform *vt, IOWriter *f)
-
VectorTransform *read_VectorTransform(const char *fname)
-
VectorTransform *read_VectorTransform(IOReader *f)
-
ProductQuantizer *read_ProductQuantizer(const char *fname)
-
ProductQuantizer *read_ProductQuantizer(IOReader *reader)
-
void write_ProductQuantizer(const ProductQuantizer *pq, const char *fname)
-
void write_ProductQuantizer(const ProductQuantizer *pq, IOWriter *f)
-
void write_InvertedLists(const InvertedLists *ils, IOWriter *f)
-
InvertedLists *read_InvertedLists(IOReader *reader, int io_flags = 0)
-
void initialize_IVFPQ_precomputed_table(int &use_precomputed_table, const Index *quantizer, const ProductQuantizer &pq, AlignedTable<float> &precomputed_table, bool by_residual, bool verbose)
Pre-compute distance tables for IVFPQ with by-residual and METRIC_L2
- Parameters:
use_precomputed_table – (I/O) =-1: force disable =0: decide heuristically (default: use tables only if they are < precomputed_tables_max_bytes), set use_precomputed_table on output =1: tables that work for all quantizers (size 256 * nlist * M) =2: specific version for MultiIndexQuantizer (much more compact)
precomputed_table – precomputed table to initialize
-
inline uint64_t lo_build(uint64_t list_id, uint64_t offset)
-
inline uint64_t lo_listno(uint64_t lo)
-
inline uint64_t lo_offset(uint64_t lo)
-
constexpr bool is_similarity_metric(MetricType metric_type)
this function is used to distinguish between min and max indexes since we need to support similarity and dis-similarity metrics in a flexible way
-
template<int A = 32>
inline bool is_aligned_pointer(const void *x)
-
float fvec_L2sqr(const float *x, const float *y, size_t d)
Squared L2 distance between two vectors.
-
float fvec_inner_product(const float *x, const float *y, size_t d)
inner product
-
float fvec_L1(const float *x, const float *y, size_t d)
L1 distance.
-
float fvec_Linf(const float *x, const float *y, size_t d)
infinity distance
-
void fvec_inner_product_batch_4(const float *x, const float *y0, const float *y1, const float *y2, const float *y3, const size_t d, float &dis0, float &dis1, float &dis2, float &dis3)
Special version of inner product that computes 4 distances between x and yi, which is performance oriented.
-
void fvec_L2sqr_batch_4(const float *x, const float *y0, const float *y1, const float *y2, const float *y3, const size_t d, float &dis0, float &dis1, float &dis2, float &dis3)
Special version of L2sqr that computes 4 distances between x and yi, which is performance oriented.
-
void pairwise_L2sqr(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)
Compute pairwise distances between sets of vectors
- Parameters:
d – dimension of the vectors
nq – nb of query vectors
nb – nb of database vectors
xq – query vectors (size nq * d)
xb – database vectors (size nb * d)
dis – output distances (size nq * nb)
ldq, ldb, ldd – strides for the matrices
-
void fvec_inner_products_ny(float *ip, const float *x, const float *y, size_t d, size_t ny)
-
void fvec_L2sqr_ny(float *dis, const float *x, const float *y, size_t d, size_t ny)
-
void fvec_L2sqr_ny_transposed(float *dis, const float *x, const float *y, const float *y_sqlen, size_t d, size_t d_offset, size_t ny)
-
size_t fvec_L2sqr_ny_nearest(float *distances_tmp_buffer, const float *x, const float *y, size_t d, size_t ny)
-
size_t fvec_L2sqr_ny_nearest_y_transposed(float *distances_tmp_buffer, const float *x, const float *y, const float *y_sqlen, size_t d, size_t d_offset, size_t ny)
-
float fvec_norm_L2sqr(const float *x, size_t d)
squared norm of a vector
-
void fvec_norms_L2(float *norms, const float *x, size_t d, size_t nx)
compute the L2 norms for a set of vectors
- Parameters:
norms – output norms, size nx
x – set of vectors, size nx * d
-
void fvec_norms_L2sqr(float *norms, const float *x, size_t d, size_t nx)
same as fvec_norms_L2, but computes squared norms
-
void fvec_renorm_L2(size_t d, size_t nx, float *x)
-
void inner_product_to_L2sqr(float *dis, const float *nr1, const float *nr2, size_t n1, size_t n2)
-
void fvec_add(size_t d, const float *a, const float *b, float *c)
compute c := a + b for vectors
c and a can overlap, c and b can overlap
- Parameters:
a – size d
b – size d
c – size d
-
void fvec_add(size_t d, const float *a, float b, float *c)
compute c := a + b for a, c vectors and b a scalar
c and a can overlap
- Parameters:
a – size d
c – size d
-
void fvec_sub(size_t d, const float *a, const float *b, float *c)
compute c := a - b for vectors
c and a can overlap, c and b can overlap
- Parameters:
a – size d
b – size d
c – size d
-
void fvec_inner_products_by_idx(float *ip, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)
compute the inner product between x and a subset y of ny vectors defined by ids
ip(i, j) = inner_product(x(i, :), y(ids(i, j), :))
- Parameters:
ip – output array, size nx * ny
x – first-term vector, size nx * d
y – second-term vector, size (max(ids) + 1) * d
ids – ids to sample from y, size nx * ny
-
void fvec_L2sqr_by_idx(float *dis, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)
compute the squared L2 distances between x and a subset y of ny vectors defined by ids
dis(i, j) = inner_product(x(i, :), y(ids(i, j), :))
- Parameters:
dis – output array, size nx * ny
x – first-term vector, size nx * d
y – second-term vector, size (max(ids) + 1) * d
ids – ids to sample from y, size nx * ny
-
void pairwise_indexed_L2sqr(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)
compute dis[j] = L2sqr(x[ix[j]], y[iy[j]]) forall j=0..n-1
- Parameters:
x – size (max(ix) + 1, d)
y – size (max(iy) + 1, d)
ix – size n
iy – size n
dis – size n
-
void pairwise_indexed_inner_product(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)
compute dis[j] = inner_product(x[ix[j]], y[iy[j]]) forall j=0..n-1
- Parameters:
x – size (max(ix) + 1, d)
y – size (max(iy) + 1, d)
ix – size n
iy – size n
dis – size n
-
void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_minheap_array_t *res, const IDSelector *sel = nullptr)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, w.r.t to max inner product.
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
res – result heap structure, which also provides k. Sorted on output
-
void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes, const IDSelector *sel = nullptr)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, for the inner product metric.
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
distances – output distances, size nq * k
indexes – output vector ids, size nq * k
-
void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_maxheap_array_t *res, const float *y_norm2 = nullptr, const IDSelector *sel = nullptr)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, for the L2 distance
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
res – result heap strcture, which also provides k. Sorted on output
y_norm2 – (optional) norms for the y vectors (nullptr or size ny)
sel – search in this subset of vectors
-
void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes, const float *y_norm2 = nullptr, const IDSelector *sel = nullptr)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, for the L2 distance
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
distances – output distances, size nq * k
indexes – output vector ids, size nq * k
y_norm2 – (optional) norms for the y vectors (nullptr or size ny)
sel – search in this subset of vectors
-
void knn_inner_products_by_idx(const float *x, const float *y, const int64_t *subset, size_t d, size_t nx, size_t ny, size_t nsubset, size_t k, float *vals, int64_t *ids, int64_t ld_ids = -1)
Find the max inner product neighbors for nx queries in a set of ny vectors indexed by ids. May be useful for re-ranking a pre-selected vector list
- Parameters:
x – query vectors, size nx * d
y – database vectors, size (max(ids) + 1) * d
ids – subset of database vectors to consider, size (nx, nsubset)
res – result structure
ld_ids – stride for the ids array. -1: use nsubset, 0: all queries process the same subset
-
void knn_L2sqr_by_idx(const float *x, const float *y, const int64_t *subset, size_t d, size_t nx, size_t ny, size_t nsubset, size_t k, float *vals, int64_t *ids, int64_t ld_subset = -1)
Find the nearest neighbors for nx queries in a set of ny vectors indexed by ids. May be useful for re-ranking a pre-selected vector list
- Parameters:
x – query vectors, size nx * d
y – database vectors, size (max(ids) + 1) * d
subset – subset of database vectors to consider, size (nx, nsubset)
res – rIDesult structure
ld_subset – stride for the subset array. -1: use nsubset, 0: all queries process the same subset
-
void range_search_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result, const IDSelector *sel = nullptr)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, w.r.t to max inner product
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
radius – search radius around the x vectors
result – result structure
-
void range_search_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result, const IDSelector *sel = nullptr)
same as range_search_L2sqr for the inner product similarity
-
void compute_PQ_dis_tables_dsub2(size_t d, size_t ksub, const float *centroids, size_t nx, const float *x, bool is_inner_product, float *dis_tables)
specialized function for PQ2
-
void fvec_madd(size_t n, const float *a, float bf, const float *b, float *c)
compute c := a + bf * b for a, b and c tables
- Parameters:
n – size of the tables
a – size n
b – size n
c – restult table, size n
-
int fvec_madd_and_argmin(size_t n, const float *a, float bf, const float *b, float *c)
same as fvec_madd, also return index of the min of the result table
- Returns:
index of the min of table c
-
bool exhaustive_L2sqr_fused_cmax(const float *x, const float *y, size_t d, size_t nx, size_t ny, Top1BlockResultHandler<CMax<float, int64_t>> &res, const float *y_norms)
-
void pairwise_extra_distances(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, MetricType mt, float metric_arg, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)
-
template<class C>
void knn_extra_metrics(const float *x, const float *y, size_t d, size_t nx, size_t ny, MetricType mt, float metric_arg, HeapArray<C> *res)
-
FlatCodesDistanceComputer *get_extra_distance_computer(size_t d, MetricType mt, float metric_arg, size_t nb, const float *xb)
get a DistanceComputer that refers to this type of distance and indexes a flat array of size nb
-
inline uint16_t encode_fp16(float x)
-
inline float decode_fp16(uint16_t x)
-
void bitvec_print(const uint8_t *b, size_t d)
-
void fvecs2bitvecs(const float *x, uint8_t *b, size_t d, size_t n)
-
void bitvecs2fvecs(const uint8_t *b, float *x, size_t d, size_t n)
-
void fvec2bitvec(const float *x, uint8_t *b, size_t d)
-
void bitvec_shuffle(size_t n, size_t da, size_t db, const int *order, const uint8_t *a, uint8_t *b)
Shuffle the bits from b(i, j) := a(i, order[j])
-
void hammings(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t nbytespercode, hamdis_t *dis)
Compute a set of Hamming distances between na and nb binary vectors
- Parameters:
a – size na * nbytespercode
b – size nb * nbytespercode
nbytespercode – should be multiple of 8
dis – output distances, size na * nb
-
void hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered, ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK)
Return the k smallest Hamming distances for a set of binary query vectors, using a max heap.
- Parameters:
a – queries, size ha->nh * ncodes
b – database, size nb * ncodes
nb – number of database vectors
ncodes – size of the binary codes (bytes)
ordered – if != 0: order the results by decreasing distance (may be bottleneck for k/n > 0.01)
approx_topk_mode – allows to use approximate top-k facilities to speedup heap
-
void hammings_knn(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered)
-
void hammings_knn_mc(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t k, size_t ncodes, int32_t *distances, int64_t *labels)
Return the k smallest Hamming distances for a set of binary query vectors, using counting max.
- Parameters:
a – queries, size na * ncodes
b – database, size nb * ncodes
na – number of query vectors
nb – number of database vectors
k – number of vectors/distances to return
ncodes – size of the binary codes (bytes)
distances – output distances from each query vector to its k nearest neighbors
labels – output ids of the k nearest neighbors to each query vector
-
void hamming_range_search(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, int radius, size_t ncodes, RangeSearchResult *result)
same as hammings_knn except we are doing a range search with radius
-
void hamming_count_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, size_t *nptr)
-
size_t match_hamming_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, int64_t *idx, hamdis_t *dis)
-
void crosshamming_count_thres(const uint8_t *dbs, size_t n, hamdis_t ht, size_t ncodes, size_t *nptr)
-
void generalized_hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t code_size, int ordered = true)
generalized Hamming distances (= count number of code bytes that are the same)
-
void pack_bitstrings(size_t n, size_t M, int nbit, const int32_t *unpacked, uint8_t *packed, size_t code_size)
Pack a set of n codes of size M * nbit
- Parameters:
n – number of codes to pack
M – number of elementary codes per code
nbit – number of bits per elementary code
unpacked – input unpacked codes, size (n, M)
packed – output packed codes, size (n, code_size)
code_size – should be >= ceil(M * nbit / 8)
-
void pack_bitstrings(size_t n, size_t M, const int32_t *nbits, const int32_t *unpacked, uint8_t *packed, size_t code_size)
Pack a set of n codes of variable sizes
- Parameters:
nbit – number of bits per entry (size M)
-
void unpack_bitstrings(size_t n, size_t M, int nbit, const uint8_t *packed, size_t code_size, int32_t *unpacked)
Unpack a set of n codes of size M * nbit
- Parameters:
n – number of codes to pack
M – number of elementary codes per code
nbit – number of bits per elementary code
unpacked – input unpacked codes, size (n, M)
packed – output packed codes, size (n, code_size)
code_size – should be >= ceil(M * nbit / 8)
-
void unpack_bitstrings(size_t n, size_t M, const int32_t *nbits, const uint8_t *packed, size_t code_size, int32_t *unpacked)
Unpack a set of n codes of variable sizes
- Parameters:
nbit – number of bits per entry (size M)
-
inline int generalized_hamming_64(uint64_t a)
-
inline int popcount32(uint32_t x)
-
inline int popcount64(uint64_t x)
- SPECIALIZED_HC (4)
- SPECIALIZED_HC (8)
- SPECIALIZED_HC (16)
- SPECIALIZED_HC (20)
- SPECIALIZED_HC (32)
- SPECIALIZED_HC (64)
-
template<class Consumer, class ...Types>
Consumer::T dispatch_HammingComputer(int code_size, Consumer &consumer, Types... args)
-
template<class C>
inline void heap_pop(size_t k, typename C::T *bh_val, typename C::TI *bh_ids) Pops the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1]. on output the element at k-1 is undefined.
-
template<class C>
inline void heap_push(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id) Pushes the element (val, ids) into the heap bh_val[0..k-2] and bh_ids[0..k-2]. on output the element at k-1 is defined.
-
template<class C>
inline void heap_replace_top(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id) Replaces the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1], and for identical bh_val[] values also sorts by bh_ids[] values.
-
template<typename T>
inline void minheap_pop(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline void minheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void minheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void maxheap_pop(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline void maxheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void maxheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<class C>
inline void heap_heapify(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x = nullptr, const typename C::TI *ids = nullptr, size_t k0 = 0)
-
template<typename T>
inline void minheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)
-
template<typename T>
inline void maxheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)
-
template<class C>
inline void heap_addn(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x, const typename C::TI *ids, size_t n)
-
template<typename T>
inline void minheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)
-
template<typename T>
inline void maxheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)
-
template<typename C>
inline size_t heap_reorder(size_t k, typename C::T *bh_val, typename C::TI *bh_ids)
-
template<typename T>
inline size_t minheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline size_t maxheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)
-
template<class C>
inline void indirect_heap_pop(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids)
-
template<class C>
inline void indirect_heap_push(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids, typename C::TI id)
-
template<class idx_t, class C>
void merge_knn_results(size_t n, size_t k, typename C::TI nshard, const typename C::T *all_distances, const idx_t *all_labels, typename C::T *distances, idx_t *labels) Merge result tables from several shards. The per-shard results are assumed to be sorted. Note that the C comparator is reversed w.r.t. the usual top-k element heap because we want the best (ie. lowest for L2) result to be on top, not the worst. Also, it needs to hold an index of a shard id (ie. usually int32 is more than enough).
- Parameters:
all_distances – size (nshard, n, k)
all_labels – size (nshard, n, k)
distances – output distances, size (n, k)
labels – output labels, size (n, k)
-
template<>
inline float cmin_nextafter<float>(float x)
-
template<>
inline float cmax_nextafter<float>(float x)
-
template<>
inline uint16_t cmin_nextafter<uint16_t>(uint16_t x)
-
template<>
inline uint16_t cmax_nextafter<uint16_t>(uint16_t x)
-
template<class C>
C::T partition_fuzzy(typename C::T *vals, typename C::TI *ids, size_t n, size_t q_min, size_t q_max, size_t *q_out) partitions the table into 0:q and q:n where all elements above q are >= all elements below q (for C = CMax, for CMin comparisons are reversed)
Returns the partition threshold. The elements q:n are destroyed on output.
-
template<class C>
inline C::T partition(typename C::T *vals, typename C::TI *ids, size_t n, size_t q) simplified interface for when the parition is not fuzzy
-
void simd_histogram_8(const uint16_t *data, int n, uint16_t min, int shift, int *hist)
low level SIMD histogramming functions 8-bin histogram of (x - min) >> shift values outside the range are ignored. the data table should be aligned on 32 bytes
-
void simd_histogram_16(const uint16_t *data, int n, uint16_t min, int shift, int *hist)
same for 16-bin histogram
-
void float_rand(float *x, size_t n, int64_t seed)
-
void float_randn(float *x, size_t n, int64_t seed)
-
void int64_rand(int64_t *x, size_t n, int64_t seed)
-
void byte_rand(uint8_t *x, size_t n, int64_t seed)
-
void int64_rand_max(int64_t *x, size_t n, uint64_t max, int64_t seed)
-
void rand_perm(int *perm, size_t n, int64_t seed)
-
void rand_smooth_vectors(size_t n, size_t d, float *x, int64_t seed)
-
inline simd16uint16 min(simd16uint16 a, simd16uint16 b)
-
inline simd16uint16 max(simd16uint16 a, simd16uint16 b)
-
inline simd16uint16 combine2x2(simd16uint16 a, simd16uint16 b)
-
inline uint32_t cmp_ge32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)
-
inline uint32_t cmp_le32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)
-
inline simd16uint16 hadd(const simd16uint16 &a, const simd16uint16 &b)
-
inline void cmplt_min_max_fast(const simd16uint16 candidateValues, const simd16uint16 candidateIndices, const simd16uint16 currentValues, const simd16uint16 currentIndices, simd16uint16 &minValues, simd16uint16 &minIndices, simd16uint16 &maxValues, simd16uint16 &maxIndices)
-
inline simd32uint8 uint16_to_uint8_saturate(simd16uint16 a, simd16uint16 b)
-
inline uint32_t get_MSBs(simd32uint8 a)
get most significant bit of each byte
-
inline simd32uint8 blendv(simd32uint8 a, simd32uint8 b, simd32uint8 mask)
use MSB of each byte of mask to select a byte between a and b
-
inline void cmplt_min_max_fast(const simd8uint32 candidateValues, const simd8uint32 candidateIndices, const simd8uint32 currentValues, const simd8uint32 currentIndices, simd8uint32 &minValues, simd8uint32 &minIndices, simd8uint32 &maxValues, simd8uint32 &maxIndices)
-
inline simd8float32 hadd(simd8float32 a, simd8float32 b)
-
inline simd8float32 unpacklo(simd8float32 a, simd8float32 b)
-
inline simd8float32 unpackhi(simd8float32 a, simd8float32 b)
-
inline simd8float32 fmadd(simd8float32 a, simd8float32 b, simd8float32 c)
-
inline void cmplt_and_blend_inplace(const simd8float32 candidateValues, const simd8uint32 candidateIndices, simd8float32 &lowestValues, simd8uint32 &lowestIndices)
-
inline void cmplt_min_max_fast(const simd8float32 candidateValues, const simd8uint32 candidateIndices, const simd8float32 currentValues, const simd8uint32 currentIndices, simd8float32 &minValues, simd8uint32 &minIndices, simd8float32 &maxValues, simd8uint32 &maxIndices)
-
inline simd16uint16 combine4x2(simd32uint16 a, simd32uint16 b)
-
inline simd16uint16 min(const simd16uint16 &av, const simd16uint16 &bv)
-
inline simd16uint16 max(const simd16uint16 &av, const simd16uint16 &bv)
-
inline simd16uint16 combine2x2(const simd16uint16 &a, const simd16uint16 &b)
-
inline uint32_t cmp_ge32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)
-
inline uint32_t cmp_le32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)
-
inline simd32uint8 uint16_to_uint8_saturate(const simd16uint16 &a, const simd16uint16 &b)
-
inline uint32_t get_MSBs(const simd32uint8 &a)
get most significant bit of each byte
-
inline simd32uint8 blendv(const simd32uint8 &a, const simd32uint8 &b, const simd32uint8 &mask)
use MSB of each byte of mask to select a byte between a and b
-
inline simd8float32 hadd(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 unpacklo(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 unpackhi(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 fmadd(const simd8float32 &a, const simd8float32 &b, const simd8float32 &c)
-
void fvec_argsort(size_t n, const float *vals, size_t *perm)
Indirect sort of a floating-point array
- Parameters:
n – size of the array
vals – array to sort, size n
perm – output: permutation of [0..n-1], st. vals[perm[i + 1]] >= vals[perm[i]]
-
void fvec_argsort_parallel(size_t n, const float *vals, size_t *perm)
Same as fvec_argsort, parallelized
-
void bucket_sort(size_t nval, const uint64_t *vals, uint64_t nbucket, int64_t *lims, int64_t *perm, int nt = 0)
Bucket sort of a list of values
- Parameters:
vals – values to sort, size nval, max value nbucket - 1
lims – output limits of buckets, size nbucket + 1
perm – output buckets, the elements of bucket i are in perm[lims[i]:lims[i + 1]]
nt – number of threads (0 = pure sequential code)
-
void matrix_bucket_sort_inplace(size_t nrow, size_t ncol, int32_t *vals, int32_t nbucket, int64_t *lims, int nt = 0)
in-place bucket sort (with attention to memory=>int32) on input the values are in a nrow * col matrix we want to store the row numbers in the output.
- Parameters:
vals – positive values to sort, size nrow * ncol, max value nbucket - 1
lims – output limits of buckets, size nbucket + 1
nt – number of threads (0 = pure sequential code)
-
void matrix_bucket_sort_inplace(size_t nrow, size_t ncol, int64_t *vals, int64_t nbucket, int64_t *lims, int nt = 0)
same with int64 elements
-
void hashtable_int64_to_int64_init(int log2_capacity, int64_t *tab)
Hashtable implementation for int64 -> int64 with external storage implemented for fast batch add and lookup.
tab is of size 2 * (1 << log2_capacity) n is the number of elements to add or search
adding several values in a same batch: an arbitrary one gets added in different batches: the newer batch overwrites. raises an exception if capacity is exhausted.
-
void hashtable_int64_to_int64_add(int log2_capacity, int64_t *tab, size_t n, const int64_t *keys, const int64_t *vals)
-
void hashtable_int64_to_int64_lookup(int log2_capacity, const int64_t *tab, size_t n, const int64_t *keys, int64_t *vals)
-
std::string get_compile_options()
get compile options
-
double getmillisecs()
ms elapsed since some arbitrary epoch
-
size_t get_mem_usage_kb()
get current RSS usage in kB
-
uint64_t get_cycles()
-
void reflection(const float *u, float *x, size_t n, size_t d, size_t nu)
-
void matrix_qr(int m, int n, float *a)
compute the Q of the QR decomposition for m > n
- Parameters:
a – size n * m: input matrix and output Q
-
void ranklist_handle_ties(int k, int64_t *idx, const float *dis)
distances are supposed to be sorted. Sorts indices with same distance
-
size_t ranklist_intersection_size(size_t k1, const int64_t *v1, size_t k2, const int64_t *v2)
count the number of common elements between v1 and v2 algorithm = sorting + bissection to avoid double-counting duplicates
-
size_t merge_result_table_with(size_t n, size_t k, int64_t *I0, float *D0, const int64_t *I1, const float *D1, bool keep_min = true, int64_t translation = 0)
merge a result table into another one
- Parameters:
I0, D0 – first result table, size (n, k)
I1, D1 – second result table, size (n, k)
keep_min – if true, keep min values, otherwise keep max
translation – add this value to all I1’s indexes
- Returns:
nb of values that were taken from the second table
-
double imbalance_factor(int n, int k, const int64_t *assign)
a balanced assignment has a IF of 1
-
double imbalance_factor(int k, const int *hist)
same, takes a histogram as input
-
int ivec_hist(size_t n, const int *v, int vmax, int *hist)
compute histogram on v
-
void bincode_hist(size_t n, size_t nbits, const uint8_t *codes, int *hist)
Compute histogram of bits on a code array
- Parameters:
codes – size(n, nbits / 8)
hist – size(nbits): nb of 1s in the array of codes
-
uint64_t ivec_checksum(size_t n, const int32_t *a)
compute a checksum on a table.
-
uint64_t bvec_checksum(size_t n, const uint8_t *a)
compute a checksum on a table.
-
void bvecs_checksum(size_t n, size_t d, const uint8_t *a, uint64_t *cs)
compute checksums for the rows of a matrix
- Parameters:
n – number of rows
d – size per row
a – matrix to handle, size n * d
cs – output checksums, size n
-
const float *fvecs_maybe_subsample(size_t d, size_t *n, size_t nmax, const float *x, bool verbose = false, int64_t seed = 1234)
random subsamples a set of vectors if there are too many of them
- Parameters:
d – dimension of the vectors
n – on input: nb of input vectors, output: nb of output vectors
nmax – max nb of vectors to keep
x – input array, size *n-by-d
seed – random seed to use for sampling
- Returns:
x or an array allocated with new [] with *n vectors
-
void binary_to_real(size_t d, const uint8_t *x_in, float *x_out)
Convert binary vector to +1/-1 valued float vector.
- Parameters:
d – dimension of the vector (multiple of 8)
x_in – input binary vector (uint8_t table of size d / 8)
x_out – output float vector (float table of size d)
-
void real_to_binary(size_t d, const float *x_in, uint8_t *x_out)
Convert float vector to binary vector. Components > 0 are converted to 1, others to 0.
- Parameters:
d – dimension of the vector (multiple of 8)
x_in – input float vector (float table of size d)
x_out – output binary vector (uint8_t table of size d / 8)
-
uint64_t hash_bytes(const uint8_t *bytes, int64_t n)
A reasonable hashing function
-
bool check_openmp()
Whether OpenMP annotations were respected.
Variables
- FAISS_API HNSWStats hnsw_stats
- FAISS_API int product_quantizer_compute_codes_bs
- FAISS_API bool simd_result_handlers_accept_virtual
- FAISS_API int index2layer_sa_encode_bs
- FAISS_API int index_factory_verbose
set to > 0 to get more logs from index_factory
-
const int IO_FLAG_READ_ONLY = 2
-
const int IO_FLAG_ONDISK_SAME_DIR = 4
-
const int IO_FLAG_SKIP_IVF_DATA = 8
-
const int IO_FLAG_SKIP_PRECOMPUTE_TABLE = 16
-
const int IO_FLAG_PQ_SKIP_SDC_TABLE = 32
-
const int IO_FLAG_MMAP = IO_FLAG_SKIP_IVF_DATA | 0x646f0000
- FAISS_API IndexBinaryHashStats indexBinaryHash_stats
- FAISS_API FastScanStats FastScan_stats
- FAISS_API bool check_compatible_for_merge_expensive_check
- FAISS_API IndexIVFStats indexIVF_stats
- FAISS_API IVFFastScanStats IVFFastScan_stats
- FAISS_API size_t precomputed_table_max_bytes
- FAISS_API int index_ivfpq_add_core_o_bs
- FAISS_API IndexIVFPQStats indexIVFPQ_stats
- FAISS_API IndexPQStats indexPQ_stats
- FAISS_API int multi_index_quantizer_search_bs
- FAISS_API int rowwise_minmax_sa_encode_bs
block size for performing sa_encode and sa_decode
- FAISS_API int rowwise_minmax_sa_decode_bs
- FAISS_API int distance_compute_blas_threshold
- FAISS_API int distance_compute_blas_query_bs
- FAISS_API int distance_compute_blas_database_bs
- FAISS_API int distance_compute_min_k_reservoir
- FAISS_API size_t hamming_batch_size
-
static constexpr uint8_t hamdis_tab_ham_bytes[256] = {0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8}
- FAISS_API PartitionStats partition_stats
- FAISS_API int bucket_sort_verbose
increase verbosity of the bucket_sort functions
-
struct AutoTuneCriterion
- #include <AutoTune.h>
Evaluation criterion. Returns a performance measure in [0,1], higher is better.
Subclassed by faiss::IntersectionCriterion, faiss::OneRecallAtRCriterion
Public Functions
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const = 0
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline virtual ~AutoTuneCriterion()
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
-
struct OneRecallAtRCriterion : public faiss::AutoTuneCriterion
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~OneRecallAtRCriterion() override
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const override
-
struct IntersectionCriterion : public faiss::AutoTuneCriterion
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~IntersectionCriterion() override
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const override
-
struct OperatingPoint
- #include <AutoTune.h>
Maintains a list of experimental results. Each operating point is a (perf, t, key) triplet, where higher perf and lower t is better. The key field is an arbitrary identifier for the operating point.
Includes primitives to extract the Pareto-optimal operating points in the (perf, t) space.
Public Members
-
double perf
performance measure (output of a Criterion)
-
double t
corresponding execution time (ms)
-
std::string key
key that identifies this op pt
-
int64_t cno
integer identifer
-
double perf
-
struct OperatingPoints
Public Functions
-
OperatingPoints()
-
int merge_with(const OperatingPoints &other, const std::string &prefix = "")
add operating points from other to this, with a prefix to the keys
-
void clear()
-
bool add(double perf, double t, const std::string &key, size_t cno = 0)
add a performance measure. Return whether it is an optimal point
-
double t_for_perf(double perf) const
get time required to obtain a given performance measure
-
void display(bool only_optimal = true) const
easy-to-read output
-
void all_to_gnuplot(const char *fname) const
output to a format easy to digest by gnuplot
-
void optimal_to_gnuplot(const char *fname) const
Public Members
-
std::vector<OperatingPoint> all_pts
all operating points
-
std::vector<OperatingPoint> optimal_pts
optimal operating points, sorted by perf
-
OperatingPoints()
-
struct ParameterRange
- #include <AutoTune.h>
possible values of a parameter, sorted from least to most expensive/accurate
-
struct ParameterSpace
- #include <AutoTune.h>
Uses a-priori knowledge on the Faiss indexes to extract tunable parameters.
Subclassed by faiss::gpu::GpuParameterSpace
Public Functions
-
ParameterSpace()
-
size_t n_combinations() const
nb of combinations, = product of values sizes
-
bool combination_ge(size_t c1, size_t c2) const
returns whether combinations c1 >= c2 in the tuple sense
-
std::string combination_name(size_t cno) const
get string representation of the combination
-
void display() const
print a description on stdout
-
ParameterRange &add_range(const std::string &name)
add a new parameter (or return it if it exists)
-
virtual void initialize(const Index *index)
initialize with reasonable parameters for the index
-
void set_index_parameters(Index *index, size_t cno) const
set a combination of parameters on an index
-
void set_index_parameters(Index *index, const char *param_string) const
set a combination of parameters described by a string
-
virtual void set_index_parameter(Index *index, const std::string &name, double val) const
set one of the parameters, returns whether setting was successful
-
void update_bounds(size_t cno, const OperatingPoint &op, double *upper_bound_perf, double *lower_bound_t) const
find an upper bound on the performance and a lower bound on t for configuration cno given another operating point op
-
void explore(Index *index, size_t nq, const float *xq, const AutoTuneCriterion &crit, OperatingPoints *ops) const
explore operating points
- Parameters:
index – index to run on
xq – query vectors (size nq * index.d)
crit – selection criterion
ops – resulting operating points
-
inline virtual ~ParameterSpace()
Public Members
-
std::vector<ParameterRange> parameter_ranges
all tunable parameters
-
int verbose
verbosity during exploration
-
int n_experiments
nb of experiments during optimization (0 = try all combinations)
-
size_t batchsize
maximum number of queries to submit at a time.
-
bool thread_over_batches
use multithreading over batches (useful to benchmark independent single-searches)
-
double min_test_duration
run tests several times until they reach at least this duration (to avoid jittering in MT mode)
-
ParameterSpace()
-
struct Cloner
- #include <clone_index.h>
Cloner class, useful to override classes with other cloning functions. The cloning function above just calls Cloner::clone_Index.
Subclassed by faiss::gpu::ToCPUCloner, faiss::gpu::ToGpuCloner, faiss::gpu::ToGpuClonerMultiple
Public Functions
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
inline virtual ~Cloner()
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
struct ClusteringParameters
- #include <Clustering.h>
Class for the clustering parameters. Can be passed to the constructor of the Clustering object.
Subclassed by faiss::Clustering, faiss::ProgressiveDimClusteringParameters
Public Members
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
int niter = 25
-
struct ClusteringIterationStats
Public Members
-
float obj
objective values (sum of distances reported by index)
-
double time
seconds for iteration
-
double time_search
seconds for just search
-
double imbalance_factor
imbalance factor of iteration
-
int nsplit
number of cluster splits
-
float obj
-
struct Clustering : public faiss::ClusteringParameters
- #include <Clustering.h>
K-means clustering based on assignment - centroid update iterations
The clustering is based on an Index object that assigns training points to the centroids. Therefore, at each iteration the centroids are added to the index.
On output, the centoids table is set to the latest version of the centroids and they are also added to the index. If the centroids table it is not empty on input, it is also used for initialization.
Subclassed by faiss::Clustering1D
Public Functions
-
Clustering(int d, int k)
-
Clustering(int d, int k, const ClusteringParameters &cp)
-
virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)
run k-means training
- Parameters:
x – training vectors, size n * d
index – index used for assignment
x_weights – weight associated to each vector: NULL or size n
-
void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)
run with encoded vectors
win addition to train()’s parameters takes a codec as parameter to decode the input vectors.
- Parameters:
codec – codec used to decode the vectors (nullptr = vectors are in fact floats)
-
void post_process_centroids()
Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding
-
inline virtual ~Clustering()
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d) if centroids are set on input to train, they will be used as initialization
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
Clustering(int d, int k)
-
struct Clustering1D : public faiss::Clustering
- #include <Clustering.h>
Exact 1D clustering algorithm
Since it does not use an index, it does not overload the train() function
Public Functions
-
explicit Clustering1D(int k)
-
Clustering1D(int k, const ClusteringParameters &cp)
-
void train_exact(idx_t n, const float *x)
-
inline virtual ~Clustering1D()
-
virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)
run k-means training
- Parameters:
x – training vectors, size n * d
index – index used for assignment
x_weights – weight associated to each vector: NULL or size n
-
void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)
run with encoded vectors
win addition to train()’s parameters takes a codec as parameter to decode the input vectors.
- Parameters:
codec – codec used to decode the vectors (nullptr = vectors are in fact floats)
-
void post_process_centroids()
Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d) if centroids are set on input to train, they will be used as initialization
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
explicit Clustering1D(int k)
-
struct ProgressiveDimClusteringParameters : public faiss::ClusteringParameters
Subclassed by faiss::ProgressiveDimClustering
Public Functions
-
ProgressiveDimClusteringParameters()
Public Members
-
int progressive_dim_steps
number of incremental steps
-
bool apply_pca
apply PCA on input
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
ProgressiveDimClusteringParameters()
-
struct ProgressiveDimIndexFactory
- #include <Clustering.h>
generates an index suitable for clustering when called
Subclassed by faiss::gpu::GpuProgressiveDimIndexFactory
Public Functions
-
virtual Index *operator()(int dim)
ownership transferred to caller
-
inline virtual ~ProgressiveDimIndexFactory()
-
virtual Index *operator()(int dim)
-
struct ProgressiveDimClustering : public faiss::ProgressiveDimClusteringParameters
- #include <Clustering.h>
K-means clustering with progressive dimensions used
The clustering first happens in dim 1, then with exponentially increasing dimension until d (I steps). This is typically applied after a PCA transformation (optional). Reference:
“Improved Residual Vector Quantization for High-dimensional Approximate
Nearest Neighbor Search”
Shicong Liu, Hongtao Lu, Junru Shao, AAAI’15
https://arxiv.org/abs/1509.05195
Public Functions
-
ProgressiveDimClustering(int d, int k)
-
ProgressiveDimClustering(int d, int k, const ProgressiveDimClusteringParameters &cp)
-
void train(idx_t n, const float *x, ProgressiveDimIndexFactory &factory)
-
inline virtual ~ProgressiveDimClustering()
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d)
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int progressive_dim_steps
number of incremental steps
-
bool apply_pca
apply PCA on input
-
int niter = 25
number of clustering iterations
-
int nredo = 1
redo clustering this many times and keep the clusters with the best objective
-
bool verbose = false
-
bool spherical = false
whether to normalize centroids after each iteration (useful for inner product clustering)
-
bool int_centroids = false
round centroids coordinates to integer after each iteration?
-
bool update_index = false
re-train index after each iteration?
-
bool frozen_centroids = false
Use the subset of centroids provided as input and do not change them during iterations
-
int min_points_per_centroid = 39
If fewer than this number of training vectors per centroid are provided, writes a warning. Note that fewer than 1 point per centroid raises an exception.
-
int max_points_per_centroid = 256
to limit size of dataset, otherwise the training set is subsampled
-
int seed = 1234
seed for the random number generator
-
size_t decode_block_size = 32768
when the training set is encoded, batch size of the codec decoder
-
ProgressiveDimClustering(int d, int k)
-
struct AdditiveQuantizer : public faiss::Quantizer
- #include <AdditiveQuantizer.h>
Abstract structure for additive quantizers
Different from the product quantizer in which the decoded vector is the concatenation of M sub-vectors, additive quantizers sum M sub-vectors to get the decoded vector.
Subclassed by faiss::LocalSearchQuantizer, faiss::ProductAdditiveQuantizer, faiss::ResidualQuantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
AdditiveQuantizer(size_t d, const std::vector<size_t> &nbits, Search_type_t search_type = ST_decompress)
-
AdditiveQuantizer()
compute derived values when d, M and nbits have been set
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const = 0
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void decode_64bit(idx_t n, float *x) const
decoding function for a code in a 64-bit word
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const
Compute inner-product look-up tables. Used in the centroid search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
-
void knn_centroids_L2(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels, const float *centroid_norms) const
Exact L2 search, with precomputed norms
-
virtual ~AdditiveQuantizer()
-
virtual void train(size_t n, const float *x) = 0
Train the quantizer
- Parameters:
x – training vectors, size n * d
Public Members
-
size_t M
number of codebooks
-
std::vector<size_t> nbits
bits for each step
-
std::vector<float> codebooks
codebooks
-
std::vector<uint64_t> codebook_offsets
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
IndexFlat1D qnorm
store and search norms
-
std::vector<float> norm_tabs
store norms of codebook entries for 4-bit fastscan search
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct RangeSearchResult
- #include <AuxIndexStructures.h>
The objective is to have a simple result structure while minimizing the number of mem copies in the result. The method do_allocation can be overloaded to allocate the result tables in the matrix type of a scripting language like Lua or Python.
Public Functions
-
explicit RangeSearchResult(size_t nq, bool alloc_lims = true)
lims must be allocated on input to range_search.
-
virtual void do_allocation()
called when lims contains the nb of elements result entries for each query
-
virtual ~RangeSearchResult()
Public Members
-
size_t nq
nb of queries
-
size_t *lims
size (nq + 1)
-
idx_t *labels
result for query i is labels[lims[i]:lims[i+1]]
-
float *distances
corresponding distances (not sorted)
-
size_t buffer_size
size of the result buffers used
-
explicit RangeSearchResult(size_t nq, bool alloc_lims = true)
-
struct BufferList
- #include <AuxIndexStructures.h>
List of temporary buffers used to store results before they are copied to the RangeSearchResult object.
Subclassed by faiss::RangeSearchPartialResult
Public Functions
-
explicit BufferList(size_t buffer_size)
-
~BufferList()
-
void append_buffer()
create a new buffer
-
void add(idx_t id, float dis)
add one result, possibly appending a new buffer if needed
-
void copy_range(size_t ofs, size_t n, idx_t *dest_ids, float *dest_dis)
copy elemnts ofs:ofs+n-1 seen as linear data in the buffers to tables dest_ids, dest_dis
Public Members
-
size_t buffer_size
-
size_t wp
write pointer in the last buffer.
-
struct Buffer
-
explicit BufferList(size_t buffer_size)
-
struct RangeQueryResult
- #include <AuxIndexStructures.h>
result structure for a single query
Public Functions
-
void add(float dis, idx_t id)
called by search function to report a new result
-
void add(float dis, idx_t id)
-
struct RangeSearchPartialResult : public faiss::BufferList
- #include <AuxIndexStructures.h>
the entries in the buffers are split per query
Public Functions
-
explicit RangeSearchPartialResult(RangeSearchResult *res_in)
eventually the result will be stored in res_in
-
RangeQueryResult &new_result(idx_t qno)
begin a new result
-
void finalize()
-
void set_lims()
called by range_search before do_allocation
-
void copy_result(bool incremental = false)
called by range_search after do_allocation
-
void append_buffer()
create a new buffer
-
void add(idx_t id, float dis)
add one result, possibly appending a new buffer if needed
-
void copy_range(size_t ofs, size_t n, idx_t *dest_ids, float *dest_dis)
copy elemnts ofs:ofs+n-1 seen as linear data in the buffers to tables dest_ids, dest_dis
Public Members
-
RangeSearchResult *res
-
std::vector<RangeQueryResult> queries
query ids + nb of results per query.
-
size_t buffer_size
-
std::vector<Buffer> buffers
-
size_t wp
write pointer in the last buffer.
Public Static Functions
-
static void merge(std::vector<RangeSearchPartialResult*> &partial_results, bool do_delete = true)
merge a set of PartialResult’s into one RangeSearchResult on ouptut the partialresults are empty!
-
explicit RangeSearchPartialResult(RangeSearchResult *res_in)
-
struct InterruptCallback
Public Functions
-
virtual bool want_interrupt() = 0
-
inline virtual ~InterruptCallback()
Public Static Functions
-
static void clear_instance()
-
static void check()
check if:
an interrupt callback is set
the callback returns true if this is the case, then throw an exception. Should not be called from multiple threads.
-
static bool is_interrupted()
same as check() but return true if is interrupted instead of throwing. Can be called from multiple threads.
-
static size_t get_period_hint(size_t flops)
assuming each iteration takes a certain number of flops, what is a reasonable interval to check for interrupts?
-
virtual bool want_interrupt() = 0
-
struct VisitedTable
- #include <AuxIndexStructures.h>
set implementation optimized for fast access.
Public Functions
-
inline explicit VisitedTable(int size)
-
inline void set(int no)
set flag #no to true
-
inline bool get(int no) const
get flag #no
-
inline void advance()
reset all flags to false
-
inline explicit VisitedTable(int size)
-
struct CodePacker
- #include <CodePacker.h>
Packing consists in combining a fixed number of codes of constant size (code_size) into a block of data where they may (or may not) be interleaved for efficient consumption by distance computation kernels. This exists for the “fast_scan” indexes on CPU and for some GPU kernels.
Subclassed by faiss::CodePackerFlat, faiss::CodePackerPQ4
Public Functions
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const = 0
-
virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const = 0
-
virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const
-
virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const
-
inline virtual ~CodePacker()
Public Members
-
size_t code_size
-
size_t nvec
-
size_t block_size
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const = 0
-
struct CodePackerFlat : public faiss::CodePacker
- #include <CodePacker.h>
Trivial code packer where codes are stored one by one
Public Functions
-
explicit CodePackerFlat(size_t code_size)
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const final
-
virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const final
-
virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const final
-
virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const final
Public Members
-
size_t code_size
-
size_t nvec
-
size_t block_size
-
explicit CodePackerFlat(size_t code_size)
-
struct DistanceComputer
Subclassed by faiss::FlatCodesDistanceComputer
Public Functions
-
virtual void set_query(const float *x) = 0
called before computing distances. Pointer x should remain valid while operator () is called
-
virtual float operator()(idx_t i) = 0
compute distance of vector i to current query
-
inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3)
compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.
-
inline virtual ~DistanceComputer()
-
virtual void set_query(const float *x) = 0
-
struct FlatCodesDistanceComputer : public faiss::DistanceComputer
Subclassed by faiss::ScalarQuantizer::SQDistanceComputer
Public Functions
-
inline FlatCodesDistanceComputer(const uint8_t *codes, size_t code_size)
-
inline FlatCodesDistanceComputer()
-
inline virtual float operator()(idx_t i) override
compute distance of vector i to current query
-
virtual float distance_to_code(const uint8_t *code) = 0
compute distance of current query to an encoded vector
-
inline virtual ~FlatCodesDistanceComputer()
-
virtual void set_query(const float *x) = 0
called before computing distances. Pointer x should remain valid while operator () is called
-
inline virtual void distances_batch_4(const idx_t idx0, const idx_t idx1, const idx_t idx2, const idx_t idx3, float &dis0, float &dis1, float &dis2, float &dis3)
compute distances of current query to 4 stored vectors. certain DistanceComputer implementations may benefit heavily from this.
Public Members
-
const uint8_t *codes
-
size_t code_size
-
inline FlatCodesDistanceComputer(const uint8_t *codes, size_t code_size)
-
class FaissException : public std::exception
- #include <FaissException.h>
Base class for Faiss exceptions.
Public Functions
-
explicit FaissException(const std::string &msg)
-
FaissException(const std::string &msg, const char *funcName, const char *file, int line)
-
const char *what() const noexcept override
from std::exception
Public Members
-
std::string msg
-
explicit FaissException(const std::string &msg)
-
struct TransformedVectors
- #include <FaissException.h>
RAII object for a set of possibly transformed vectors (deallocated only if they are indeed transformed)
Public Functions
-
inline TransformedVectors(const float *x_orig, const float *x)
-
inline ~TransformedVectors()
Public Members
-
const float *x
-
bool own_x
-
inline TransformedVectors(const float *x_orig, const float *x)
-
template<class C>
struct ResultHandler Subclassed by faiss::HeapBlockResultHandler< C >::SingleResultHandler, faiss::RangeSearchBlockResultHandler< C >::SingleResultHandler, faiss::ReservoirTopN< C >, faiss::Top1BlockResultHandler< C >::SingleResultHandler
Public Functions
-
inline virtual ~ResultHandler()
Public Members
-
C::T threshold = 0
-
inline virtual ~ResultHandler()
-
struct SearchParametersHNSW : public faiss::SearchParameters
Public Functions
-
inline ~SearchParametersHNSW()
Public Members
-
int efSearch = 16
-
bool check_relative_distance = true
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline ~SearchParametersHNSW()
-
struct HNSW
Public Types
-
using storage_idx_t = int32_t
internal storage of vectors (32 bits: this is expensive)
-
using C = CMax<float, int64_t>
-
typedef std::pair<float, storage_idx_t> Node
Public Functions
-
void set_default_probas(int M, float levelMult)
initialize the assign_probas and cum_nneighbor_per_level to have 2*M links on level 0 and M links on levels > 0
-
void set_nb_neighbors(int level_no, int n)
set nb of neighbors for this level (before adding anything)
-
int nb_neighbors(int layer_no) const
nb of neighbors for this level
-
int cum_nb_neighbors(int layer_no) const
cumumlative nb up to (and excluding) this level
-
void neighbor_range(idx_t no, int layer_no, size_t *begin, size_t *end) const
range of entries in the neighbors table of vertex no at layer_no
-
explicit HNSW(int M = 32)
only mandatory parameter: nb of neighbors
-
int random_level()
pick a random level for a new point
-
void fill_with_random_links(size_t n)
add n random levels to table (for debugging…)
-
void add_links_starting_from(DistanceComputer &ptdis, storage_idx_t pt_id, storage_idx_t nearest, float d_nearest, int level, omp_lock_t *locks, VisitedTable &vt)
-
void add_with_locks(DistanceComputer &ptdis, int pt_level, int pt_id, std::vector<omp_lock_t> &locks, VisitedTable &vt)
add point pt_id on all levels <= pt_level and build the link structure for them.
-
HNSWStats search(DistanceComputer &qdis, ResultHandler<C> &res, VisitedTable &vt, const SearchParametersHNSW *params = nullptr) const
search interface for 1 point, single thread
-
void search_level_0(DistanceComputer &qdis, ResultHandler<C> &res, idx_t nprobe, const storage_idx_t *nearest_i, const float *nearest_d, int search_type, HNSWStats &search_stats, VisitedTable &vt) const
search only in level 0 from a given vertex
-
void reset()
-
void clear_neighbor_tables(int level)
-
void print_neighbor_stats(int level) const
-
int prepare_level_tab(size_t n, bool preset_levels = false)
-
void permute_entries(const idx_t *map)
Public Members
-
std::vector<double> assign_probas
assignment probability to each layer (sum=1)
-
std::vector<int> cum_nneighbor_per_level
number of neighbors stored per layer (cumulative), should not be changed after first add
-
std::vector<int> levels
level of each vector (base level = 1), size = ntotal
-
std::vector<size_t> offsets
offsets[i] is the offset in the neighbors array where vector i is stored size ntotal + 1
-
std::vector<storage_idx_t> neighbors
neighbors[offsets[i]:offsets[i+1]] is the list of neighbors of vector i for all levels. this is where all storage goes.
-
storage_idx_t entry_point = -1
entry point in the search structure (one of the points with maximum level
-
faiss::RandomGenerator rng
-
int max_level = -1
maximum level
-
int efConstruction = 40
expansion factor at construction time
-
int efSearch = 16
expansion factor at search time
-
bool check_relative_distance = true
during search: do we check whether the next best distance is good enough?
-
int upper_beam = 1
number of entry points in levels > 0.
-
bool search_bounded_queue = true
use bounded queue during exploration
Public Static Functions
-
static void shrink_neighbor_list(DistanceComputer &qdis, std::priority_queue<NodeDistFarther> &input, std::vector<NodeDistFarther> &output, int max_size)
-
struct MinimaxHeap
- #include <HNSW.h>
Heap structure that allows fast
Public Types
-
typedef faiss::CMax<float, storage_idx_t> HC
Public Functions
-
inline explicit MinimaxHeap(int n)
-
void push(storage_idx_t i, float v)
-
float max() const
-
int size() const
-
void clear()
-
int pop_min(float *vmin_out = nullptr)
-
int count_below(float thresh)
-
typedef faiss::CMax<float, storage_idx_t> HC
-
struct NodeDistCloser
- #include <HNSW.h>
to sort pairs of (id, distance) from nearest to fathest or the reverse
Public Functions
-
inline NodeDistCloser(float d, int id)
-
inline bool operator<(const NodeDistCloser &obj1) const
Public Members
-
float d
-
int id
-
inline NodeDistCloser(float d, int id)
-
struct NodeDistFarther
Public Functions
-
inline NodeDistFarther(float d, int id)
-
inline bool operator<(const NodeDistFarther &obj1) const
Public Members
-
float d
-
int id
-
inline NodeDistFarther(float d, int id)
-
using storage_idx_t = int32_t
-
struct HNSWStats
Public Functions
-
inline void reset()
number of distances computed
-
inline void combine(const HNSWStats &other)
Public Members
-
size_t n1 = 0
-
size_t n2 = 0
numbner of vectors searched
-
size_t ndis = 0
number of queries for which the candidate list is exhasted
-
inline void reset()
-
struct IDSelector
- #include <IDSelector.h>
Encapsulates a set of ids to handle.
Subclassed by PyCallbackIDSelector, faiss::IDSelectorAll, faiss::IDSelectorAnd, faiss::IDSelectorArray, faiss::IDSelectorBatch, faiss::IDSelectorBitmap, faiss::IDSelectorNot, faiss::IDSelectorOr, faiss::IDSelectorRange, faiss::IDSelectorTranslated, faiss::IDSelectorXOr
-
struct IDSelectorRange : public faiss::IDSelector
- #include <IDSelector.h>
ids between [imin, imax)
-
struct IDSelectorArray : public faiss::IDSelector
- #include <IDSelector.h>
Simple array of elements
is_member calls are very inefficient, but some operations can use the ids directly.
Public Functions
-
IDSelectorArray(size_t n, const idx_t *ids)
Construct with an array of ids to process
- Parameters:
n – number of ids to store
ids – elements to store. The pointer should remain valid during IDSelectorArray’s lifetime
-
virtual bool is_member(idx_t id) const final
-
inline ~IDSelectorArray() override
-
IDSelectorArray(size_t n, const idx_t *ids)
-
struct IDSelectorBatch : public faiss::IDSelector
- #include <IDSelector.h>
Ids from a set.
Repetitions of ids in the indices set passed to the constructor does not hurt performance.
The hash function used for the bloom filter and GCC’s implementation of unordered_set are just the least significant bits of the id. This works fine for random ids or ids in sequences but will produce many hash collisions if lsb’s are always the same
-
struct IDSelectorBitmap : public faiss::IDSelector
- #include <IDSelector.h>
One bit per element. Constructed with a bitmap, size ceil(n / 8).
Public Functions
-
IDSelectorBitmap(size_t n, const uint8_t *bitmap)
Construct with a binary mask
- Parameters:
n – size of the bitmap array
bitmap – id will be selected iff id / 8 < n and bit number (i%8) of bitmap[floor(i / 8)] is 1.
-
virtual bool is_member(idx_t id) const final
-
inline ~IDSelectorBitmap() override
Public Members
-
size_t n
-
const uint8_t *bitmap
-
IDSelectorBitmap(size_t n, const uint8_t *bitmap)
-
struct IDSelectorNot : public faiss::IDSelector
- #include <IDSelector.h>
reverts the membership test of another selector
Public Functions
-
inline IDSelectorNot(const IDSelector *sel)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorNot()
Public Members
-
const IDSelector *sel
-
inline IDSelectorNot(const IDSelector *sel)
-
struct IDSelectorAll : public faiss::IDSelector
- #include <IDSelector.h>
selects all entries (useful for benchmarking)
Public Functions
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorAll()
-
inline virtual bool is_member(idx_t id) const final
-
struct IDSelectorAnd : public faiss::IDSelector
- #include <IDSelector.h>
does an AND operation on the the two given IDSelector’s is_membership results.
Public Functions
-
inline IDSelectorAnd(const IDSelector *lhs, const IDSelector *rhs)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorAnd()
-
inline IDSelectorAnd(const IDSelector *lhs, const IDSelector *rhs)
-
struct IDSelectorOr : public faiss::IDSelector
- #include <IDSelector.h>
does an OR operation on the the two given IDSelector’s is_membership results.
Public Functions
-
inline IDSelectorOr(const IDSelector *lhs, const IDSelector *rhs)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorOr()
-
inline IDSelectorOr(const IDSelector *lhs, const IDSelector *rhs)
-
struct IDSelectorXOr : public faiss::IDSelector
- #include <IDSelector.h>
does an XOR operation on the the two given IDSelector’s is_membership results.
Public Functions
-
inline IDSelectorXOr(const IDSelector *lhs, const IDSelector *rhs)
-
inline virtual bool is_member(idx_t id) const final
-
inline virtual ~IDSelectorXOr()
-
inline IDSelectorXOr(const IDSelector *lhs, const IDSelector *rhs)
-
struct IOReader
Subclassed by PyCallbackIOReader, faiss::BufferedIOReader, faiss::FileIOReader, faiss::VectorIOReader
Public Functions
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) = 0
-
virtual int filedescriptor()
-
inline virtual ~IOReader()
Public Members
-
std::string name
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) = 0
-
struct IOWriter
Subclassed by PyCallbackIOWriter, faiss::BufferedIOWriter, faiss::FileIOWriter, faiss::VectorIOWriter
Public Functions
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) = 0
-
virtual int filedescriptor()
-
inline virtual ~IOWriter() noexcept(false)
Public Members
-
std::string name
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) = 0
-
struct VectorIOReader : public faiss::IOReader
Public Functions
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor()
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
struct VectorIOWriter : public faiss::IOWriter
Public Functions
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor()
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
struct FileIOReader : public faiss::IOReader
Public Functions
-
FileIOReader(FILE *rf)
-
FileIOReader(const char *fname)
-
~FileIOReader() override
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor() override
-
FileIOReader(FILE *rf)
-
struct FileIOWriter : public faiss::IOWriter
Public Functions
-
FileIOWriter(FILE *wf)
-
FileIOWriter(const char *fname)
-
~FileIOWriter() override
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor() override
-
FileIOWriter(FILE *wf)
-
struct BufferedIOReader : public faiss::IOReader
- #include <io.h>
wraps an ioreader to make buffered reads to avoid too small reads
Public Functions
-
explicit BufferedIOReader(IOReader *reader, size_t bsz = 1024 * 1024)
- Parameters:
bsz – buffer size (bytes). Reads will be done by batched of this size
-
virtual size_t operator()(void *ptr, size_t size, size_t nitems) override
-
virtual int filedescriptor()
-
explicit BufferedIOReader(IOReader *reader, size_t bsz = 1024 * 1024)
-
struct BufferedIOWriter : public faiss::IOWriter
Public Functions
-
explicit BufferedIOWriter(IOWriter *writer, size_t bsz = 1024 * 1024)
-
virtual size_t operator()(const void *ptr, size_t size, size_t nitems) override
-
~BufferedIOWriter() override
-
virtual int filedescriptor()
-
explicit BufferedIOWriter(IOWriter *writer, size_t bsz = 1024 * 1024)
-
struct ZnSphereSearch
- #include <lattice_Zn.h>
returns the nearest vertex in the sphere to a query. Returns only the coordinates, not an id.
Algorithm: all points are derived from a one atom vector up to a permutation and sign changes. The search function finds the most appropriate atom and transformation.
Subclassed by faiss::ZnSphereCodec
Public Functions
-
ZnSphereSearch(int dim, int r2)
-
float search(const float *x, float *c) const
find nearest centroid. x does not need to be normalized
-
float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const
full call. Requires externally-allocated temp space
-
void search_multi(int n, const float *x, float *c_out, float *dp_out)
-
ZnSphereSearch(int dim, int r2)
-
struct EnumeratedVectors
Subclassed by faiss::ZnSphereCodec, faiss::ZnSphereCodecRec
Public Functions
-
inline explicit EnumeratedVectors(int dim)
-
virtual uint64_t encode(const float *x) const = 0
encode a vector from a collection
-
virtual void decode(uint64_t code, float *c) const = 0
decode it
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
-
inline virtual ~EnumeratedVectors()
Public Members
-
uint64_t nv
size of the collection
-
int dim
-
inline explicit EnumeratedVectors(int dim)
-
struct Repeat
Public Members
-
float val
-
int n
-
float val
-
struct Repeats
- #include <lattice_Zn.h>
Repeats: used to encode a vector that has n occurrences of val. Encodes the signs and permutation of the vector. Useful for atoms.
Subclassed by faiss::ZnSphereCodec::CodeSegment
Public Functions
-
Repeats(int dim = 0, const float *c = nullptr)
-
uint64_t count() const
-
uint64_t encode(const float *c) const
-
void decode(uint64_t code, float *c) const
-
Repeats(int dim = 0, const float *c = nullptr)
-
struct ZnSphereCodec : public faiss::ZnSphereSearch, public faiss::EnumeratedVectors
- #include <lattice_Zn.h>
codec that can return ids for the encoded vectors
uses the ZnSphereSearch to encode the vector by encoding the permutation and signs. Depends on ZnSphereSearch because it uses the atom numbers
Subclassed by faiss::ZnSphereCodecAlt
Public Functions
-
ZnSphereCodec(int dim, int r2)
-
uint64_t search_and_encode(const float *x) const
-
virtual void decode(uint64_t code, float *c) const override
decode it
-
virtual uint64_t encode(const float *x) const override
takes vectors that do not need to be centroids
-
float search(const float *x, float *c) const
find nearest centroid. x does not need to be normalized
-
float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const
full call. Requires externally-allocated temp space
-
void search_multi(int n, const float *x, float *c_out, float *dp_out)
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
Public Members
-
std::vector<CodeSegment> code_segments
-
uint64_t nv
-
size_t code_size
-
int dimS
-
int r2
-
int natom
-
std::vector<float> voc
size dim * ntatom
-
int dim
-
ZnSphereCodec(int dim, int r2)
-
struct ZnSphereCodecRec : public faiss::EnumeratedVectors
- #include <lattice_Zn.h>
recursive sphere codec
Uses a recursive decomposition on the dimensions to encode centroids found by the ZnSphereSearch. The codes are not compatible with the ones of ZnSpehreCodec
Public Functions
-
ZnSphereCodecRec(int dim, int r2)
-
uint64_t encode_centroid(const float *c) const
-
virtual void decode(uint64_t code, float *c) const override
decode it
-
virtual uint64_t encode(const float *x) const override
vectors need to be centroids (does not work on arbitrary vectors)
-
uint64_t get_nv(int ld, int r2a) const
-
uint64_t get_nv_cum(int ld, int r2t, int r2a) const
-
void set_nv_cum(int ld, int r2t, int r2a, uint64_t v)
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
-
ZnSphereCodecRec(int dim, int r2)
-
struct ZnSphereCodecAlt : public faiss::ZnSphereCodec
- #include <lattice_Zn.h>
Codec that uses the recursive codec if dim is a power of 2 and the regular one otherwise
Public Functions
-
ZnSphereCodecAlt(int dim, int r2)
-
virtual uint64_t encode(const float *x) const override
takes vectors that do not need to be centroids
-
virtual void decode(uint64_t code, float *c) const override
decode it
-
uint64_t search_and_encode(const float *x) const
-
float search(const float *x, float *c) const
find nearest centroid. x does not need to be normalized
-
float search(const float *x, float *c, float *tmp, int *tmp_int, int *ibest_out = nullptr) const
full call. Requires externally-allocated temp space
-
void search_multi(int n, const float *x, float *c_out, float *dp_out)
-
void encode_multi(size_t nc, const float *c, uint64_t *codes) const
-
void decode_multi(size_t nc, const uint64_t *codes, float *c) const
-
void find_nn(size_t n, const uint64_t *codes, size_t nq, const float *xq, int64_t *idx, float *dis)
Public Members
-
bool use_rec
-
ZnSphereCodecRec znc_rec
-
std::vector<CodeSegment> code_segments
-
uint64_t nv
-
size_t code_size
-
int dimS
-
int r2
-
int natom
-
std::vector<float> voc
size dim * ntatom
-
int dim
-
ZnSphereCodecAlt(int dim, int r2)
-
struct LocalSearchQuantizer : public faiss::AdditiveQuantizer
- #include <LocalSearchQuantizer.h>
Implementation of LSQ/LSQ++ described in the following two papers:
Revisiting additive quantization Julieta Martinez, et al. ECCV 2016
LSQ++: Lower running time and higher recall in multi-codebook quantization Julieta Martinez, et al. ECCV 2018
This implementation is mostly translated from the Julia implementations by Julieta Martinez: (https://github.com/una-dinosauria/local-search-quantization, https://github.com/una-dinosauria/Rayuela.jl)
The trained codes are stored in
codebookswhich is calledcentroidsin PQ and RQ.Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
LocalSearchQuantizer(size_t d, size_t M, size_t nbits, Search_type_t search_type = ST_decompress)
-
LocalSearchQuantizer()
-
~LocalSearchQuantizer() override
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
n – number of vectors
centroids – centroids to be added to x, size n * d
-
void update_codebooks(const float *x, const int32_t *codes, size_t n)
Update codebooks given encodings
- Parameters:
x – training vectors, size n * d
codes – encoded training vectors, size n * M
n – number of vectors
-
void icm_encode(int32_t *codes, const float *x, size_t n, size_t ils_iters, std::mt19937 &gen) const
Encode vectors given codebooks using iterative conditional mode (icm).
- Parameters:
codes – output codes, size n * M
x – vectors to encode, size n * d
n – number of vectors
ils_iters – number of iterations of iterative local search
-
void icm_encode_impl(int32_t *codes, const float *x, const float *unaries, std::mt19937 &gen, size_t n, size_t ils_iters, bool verbose) const
-
void icm_encode_step(int32_t *codes, const float *unaries, const float *binaries, size_t n, size_t n_iters) const
-
void perturb_codes(int32_t *codes, size_t n, std::mt19937 &gen) const
Add some perturbation to codes
- Parameters:
codes – codes to be perturbed, size n * M
n – number of vectors
-
void perturb_codebooks(float T, const std::vector<float> &stddev, std::mt19937 &gen)
Add some perturbation to codebooks
- Parameters:
T – temperature of simulated annealing
stddev – standard derivations of each dimension in training data
-
void compute_binary_terms(float *binaries) const
Compute binary terms
- Parameters:
binaries – binary terms, size M * M * K * K
-
void compute_unary_terms(const float *x, float *unaries, size_t n) const
Compute unary terms
- Parameters:
n – number of vectors
x – vectors to encode, size n * d
unaries – unary terms, size n * M * K
-
float evaluate(const int32_t *codes, const float *x, size_t n, float *objs = nullptr) const
Helper function to compute reconstruction error
- Parameters:
codes – encoded codes, size n * M
x – vectors to encode, size n * d
n – number of vectors
objs – if it is not null, store reconstruction error of each vector into it, size n
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const
Compute inner-product look-up tables. Used in the centroid search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t K
number of codes per codebook
-
size_t train_iters = 25
number of iterations in training
-
size_t encode_ils_iters = 16
iterations of local search in encoding
-
size_t train_ils_iters = 8
iterations of local search in training
-
size_t icm_iters = 4
number of iterations in icm
-
float p = 0.5f
temperature factor
-
float lambd = 1e-2f
regularization factor
-
size_t chunk_size = 10000
nb of vectors to encode at a time
-
int random_seed = 0x12345
seed for random generator
-
size_t nperts = 4
number of perturbation in each code
if non-NULL, use this encoder to encode (owned by the object)
-
lsq::IcmEncoderFactory *icm_encoder_factory = nullptr
-
bool update_codebooks_with_double = true
-
size_t M
number of codebooks
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
IndexFlat1D qnorm
store and search norms
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct DummyScaler
- #include <LookupTableScaler.h>
no-op handler
Public Functions
-
inline simd32uint8 lookup(const simd32uint8&, const simd32uint8&) const
-
inline simd16uint16 scale_lo(const simd32uint8&) const
-
inline simd16uint16 scale_hi(const simd32uint8&) const
Public Static Attributes
-
static constexpr int nscale = 0
-
inline simd32uint8 lookup(const simd32uint8&, const simd32uint8&) const
-
struct NormTableScaler
- #include <LookupTableScaler.h>
consumes 2x4 bits to encode a norm as a scalar additive quantizer the norm is scaled because its range if larger than other components
Public Functions
-
inline explicit NormTableScaler(int scale)
-
inline simd32uint8 lookup(const simd32uint8 &lut, const simd32uint8 &c) const
-
inline simd16uint16 scale_lo(const simd32uint8 &res) const
-
inline simd16uint16 scale_hi(const simd32uint8 &res) const
Public Static Attributes
-
static constexpr int nscale = 2
-
inline explicit NormTableScaler(int scale)
-
struct NNDescent
-
Public Functions
-
explicit NNDescent(const int d, const int K)
-
~NNDescent()
-
void build(DistanceComputer &qdis, const int n, bool verbose)
-
void search(DistanceComputer &qdis, const int topk, idx_t *indices, float *dists, VisitedTable &vt) const
-
void reset()
-
void init_graph(DistanceComputer &qdis)
Initialize the KNN graph randomly.
-
void nndescent(DistanceComputer &qdis, bool verbose)
Perform NNDescent algorithm.
-
void join(DistanceComputer &qdis)
Perform local join on each node.
-
void update()
Sample new neighbors for each node to peform local join later.
-
void generate_eval_set(DistanceComputer &qdis, std::vector<int> &c, std::vector<std::vector<int>> &v, int N)
Sample a small number of points to evaluate the quality of KNNG built.
-
explicit NNDescent(const int d, const int K)
-
struct NSG
Public Types
-
using storage_idx_t = int32_t
internal storage of vectors (32 bits: this is expensive)
Public Functions
-
explicit NSG(int R = 32)
-
void reset()
-
void search(DistanceComputer &dis, int k, idx_t *I, float *D, VisitedTable &vt) const
-
template<bool collect_fullset, class index_t>
void search_on_graph(const nsg::Graph<index_t> &graph, DistanceComputer &dis, VisitedTable &vt, int ep, int pool_size, std::vector<Neighbor> &retset, std::vector<Node> &fullset) const
-
void add_reverse_links(int q, std::vector<std::mutex> &locks, DistanceComputer &dis, nsg::Graph<Node> &graph)
-
void sync_prune(int q, std::vector<Node> &pool, DistanceComputer &dis, VisitedTable &vt, const nsg::Graph<idx_t> &knn_graph, nsg::Graph<Node> &graph)
-
void link(Index *storage, const nsg::Graph<idx_t> &knn_graph, nsg::Graph<Node> &graph, bool verbose)
-
int dfs(VisitedTable &vt, int root, int cnt) const
-
int attach_unlinked(Index *storage, VisitedTable &vt, VisitedTable &vt2, std::vector<int> °rees)
-
void check_graph() const
Public Members
-
int ntotal = 0
nb of nodes
-
int R
nb of neighbors per node
-
int L
length of the search path at construction time
-
int C
candidate pool size at construction time
-
int search_L = 16
length of the search path
-
int enterpoint
enterpoint
-
bool is_built = false
NSG is built or not.
-
RandomGenerator rng
random generator
-
using storage_idx_t = int32_t
-
struct SimulatedAnnealingParameters
- #include <PolysemousTraining.h>
parameters used for the simulated annealing method
Subclassed by faiss::PolysemousTraining, faiss::SimulatedAnnealingOptimizer
Public Functions
-
inline SimulatedAnnealingParameters()
Public Members
-
double init_temperature = 0.7
-
double temperature_decay = 0.9997893011688015
-
int n_iter = 500000
-
int n_redo = 2
-
int seed = 123
-
int verbose = 0
-
bool only_bit_flips = false
-
bool init_random = false
-
inline SimulatedAnnealingParameters()
-
struct PermutationObjective
- #include <PolysemousTraining.h>
abstract class for the loss function
Subclassed by faiss::ReproduceDistancesObjective
Public Functions
-
virtual double compute_cost(const int *perm) const = 0
-
virtual double cost_update(const int *perm, int iw, int jw) const
-
inline virtual ~PermutationObjective()
Public Members
-
int n
-
virtual double compute_cost(const int *perm) const = 0
-
struct ReproduceDistancesObjective : public faiss::PermutationObjective
Public Functions
-
double dis_weight(double x) const
-
double get_source_dis(int i, int j) const
-
virtual double compute_cost(const int *perm) const override
-
virtual double cost_update(const int *perm, int iw, int jw) const override
-
ReproduceDistancesObjective(int n, const double *source_dis_in, const double *target_dis_in, double dis_weight_factor)
-
void set_affine_target_dis(const double *source_dis_in)
-
inline ~ReproduceDistancesObjective() override
Public Members
-
double dis_weight_factor
-
std::vector<double> source_dis
“real” corrected distances (size n^2)
-
const double *target_dis
wanted distances (size n^2)
-
std::vector<double> weights
weights for each distance (size n^2)
-
int n
Public Static Functions
-
static inline double sqr(double x)
-
static void compute_mean_stdev(const double *tab, size_t n2, double *mean_out, double *stddev_out)
-
double dis_weight(double x) const
-
struct SimulatedAnnealingOptimizer : public faiss::SimulatedAnnealingParameters
- #include <PolysemousTraining.h>
Simulated annealing optimization algorithm for permutations.
Public Functions
-
SimulatedAnnealingOptimizer(PermutationObjective *obj, const SimulatedAnnealingParameters &p)
logs values of the cost function
-
double optimize(int *perm)
-
double run_optimization(int *best_perm)
-
virtual ~SimulatedAnnealingOptimizer()
Public Members
-
PermutationObjective *obj
-
int n
size of the permutation
-
FILE *logfile
-
RandomGenerator *rnd
-
double init_cost
remember initial cost of optimization
-
double init_temperature = 0.7
-
double temperature_decay = 0.9997893011688015
-
int n_iter = 500000
-
int n_redo = 2
-
int seed = 123
-
int verbose = 0
-
bool only_bit_flips = false
-
bool init_random = false
-
SimulatedAnnealingOptimizer(PermutationObjective *obj, const SimulatedAnnealingParameters &p)
-
struct PolysemousTraining : public faiss::SimulatedAnnealingParameters
- #include <PolysemousTraining.h>
optimizes the order of indices in a ProductQuantizer
Public Types
-
enum Optimization_type_t
Values:
-
enumerator OT_None
-
enumerator OT_ReproduceDistances_affine
default
-
enumerator OT_Ranking_weighted_diff
same as _2, but use rank of y+ - rank of y-
-
enumerator OT_None
Public Functions
-
PolysemousTraining()
-
void optimize_pq_for_hamming(ProductQuantizer &pq, size_t n, const float *x) const
reorder the centroids so that the Hamming distance becomes a good approximation of the SDC distance (called by train)
-
void optimize_ranking(ProductQuantizer &pq, size_t n, const float *x) const
called by optimize_pq_for_hamming
-
void optimize_reproduce_distances(ProductQuantizer &pq) const
called by optimize_pq_for_hamming
-
size_t memory_usage_per_thread(const ProductQuantizer &pq) const
make sure we don’t blow up the memory
Public Members
-
Optimization_type_t optimization_type
-
int ntrain_permutation
use 1/4 of the training points for the optimization, with max. ntrain_permutation. If ntrain_permutation == 0: train on centroids
-
double dis_weight_factor
decay of exp that weights distance loss
-
size_t max_memory
refuse to train if it would require more than that amount of RAM
-
std::string log_pattern
-
double init_temperature = 0.7
-
double temperature_decay = 0.9997893011688015
-
int n_iter = 500000
-
int n_redo = 2
-
int seed = 123
-
int verbose = 0
-
bool only_bit_flips = false
-
bool init_random = false
-
enum Optimization_type_t
-
struct CodePackerPQ4 : public faiss::CodePacker
- #include <pq4_fast_scan.h>
CodePacker API for the PQ4 fast-scan
Public Functions
-
CodePackerPQ4(size_t nsq, size_t bbs)
-
virtual void pack_1(const uint8_t *flat_code, size_t offset, uint8_t *block) const final
-
virtual void unpack_1(const uint8_t *block, size_t offset, uint8_t *flat_code) const final
-
virtual void pack_all(const uint8_t *flat_codes, uint8_t *block) const
-
virtual void unpack_all(const uint8_t *block, uint8_t *flat_codes) const
-
CodePackerPQ4(size_t nsq, size_t bbs)
-
struct ProductAdditiveQuantizer : public faiss::AdditiveQuantizer
- #include <ProductAdditiveQuantizer.h>
Product Additive Quantizers
The product additive quantizer is a variant of AQ and PQ. It first splits the vector space into multiple orthogonal sub-spaces just like PQ does. And then it quantizes each sub-space by an independent additive quantizer.
Subclassed by faiss::ProductLocalSearchQuantizer, faiss::ProductResidualQuantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ProductAdditiveQuantizer(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type = ST_decompress)
Construct a product additive quantizer.
The additive quantizers passed in will be cloned into the ProductAdditiveQuantizer object.
- Parameters:
d – dimensionality of the input vectors
aqs – sub-additive quantizers
search_type – AQ search type
-
ProductAdditiveQuantizer()
-
virtual ~ProductAdditiveQuantizer()
-
void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)
-
AdditiveQuantizer *subquantizer(size_t m) const
Train the product additive quantizer.
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override
Compute inner-product look-up tables. Used in the search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t nsplits
number of sub-vectors we split a vector into
-
std::vector<AdditiveQuantizer*> quantizers
-
size_t M
number of codebooks
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
IndexFlat1D qnorm
store and search norms
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct ProductLocalSearchQuantizer : public faiss::ProductAdditiveQuantizer
- #include <ProductAdditiveQuantizer.h>
Product Local Search Quantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ProductLocalSearchQuantizer(size_t d, size_t nsplits, size_t Msub, size_t nbits, Search_type_t search_type = ST_decompress)
Construct a product LSQ object.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of sub-vectors we split a vector into
Msub – number of codebooks of each LSQ
nbits – bits for each step
search_type – AQ search type
-
ProductLocalSearchQuantizer()
-
void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)
-
AdditiveQuantizer *subquantizer(size_t m) const
Train the product additive quantizer.
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override
Compute inner-product look-up tables. Used in the search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t nsplits
number of sub-vectors we split a vector into
-
std::vector<AdditiveQuantizer*> quantizers
-
size_t M
number of codebooks
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
IndexFlat1D qnorm
store and search norms
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct ProductResidualQuantizer : public faiss::ProductAdditiveQuantizer
- #include <ProductAdditiveQuantizer.h>
Product Residual Quantizer
Public Types
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ProductResidualQuantizer(size_t d, size_t nsplits, size_t Msub, size_t nbits, Search_type_t search_type = ST_decompress)
Construct a product RQ object.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of sub-vectors we split a vector into
Msub – number of codebooks of each RQ
nbits – bits for each step
search_type – AQ search type
-
ProductResidualQuantizer()
-
void init(size_t d, const std::vector<AdditiveQuantizer*> &aqs, Search_type_t search_type)
-
AdditiveQuantizer *subquantizer(size_t m) const
Train the product additive quantizer.
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void compute_unpacked_codes(const float *x, int32_t *codes, size_t n, const float *centroids = nullptr) const
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const override
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const override
Compute inner-product look-up tables. Used in the search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
size_t nsplits
number of sub-vectors we split a vector into
-
std::vector<AdditiveQuantizer*> quantizers
-
size_t M
number of codebooks
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
IndexFlat1D qnorm
store and search norms
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum Search_type_t
-
struct ProductQuantizer : public faiss::Quantizer
- #include <ProductQuantizer.h>
Product Quantizer. Implemented only for METRIC_L2
Public Types
-
enum train_type_t
initialization
Values:
-
enumerator Train_default
-
enumerator Train_hot_start
the centroids are already initialized
share dictionary across PQ segments
-
enumerator Train_hypercube
initialize centroids with nbits-D hypercube
-
enumerator Train_hypercube_pca
initialize centroids with nbits-D hypercube
-
enumerator Train_default
Public Functions
-
inline float *get_centroids(size_t m, size_t i)
return the centroids associated with subvector m
-
inline const float *get_centroids(size_t m, size_t i) const
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
ProductQuantizer(size_t d, size_t M, size_t nbits)
-
ProductQuantizer()
-
void set_derived_values()
compute derived values when d, M and nbits have been set
-
void set_params(const float *centroids, int m)
Define the centroids for subquantizer m.
-
void compute_code(const float *x, uint8_t *code) const
Quantize one vector with the product quantizer.
-
virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
same as compute_code for several vectors
-
void compute_codes_with_assign_index(const float *x, uint8_t *codes, size_t n)
speed up code assignment using assign_index (non-const because the index is changed)
-
void decode(const uint8_t *code, float *x) const
decode a vector from a given code (or n vectors if third argument)
-
virtual void decode(const uint8_t *code, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – input codes, size n * code_size
x – output vectors, size n * d
-
void compute_code_from_distance_table(const float *tab, uint8_t *code) const
If we happen to have the distance tables precomputed, this is more efficient to compute the codes.
-
void compute_distance_table(const float *x, float *dis_table) const
Compute distance table for one vector.
The distance table for x = [x_0 x_1 .. x_(M-1)] is a M * ksub matrix that contains
dis_table (m, j) = || x_m - c_(m, j)||^2 for m = 0..M-1 and j = 0 .. ksub - 1
where c_(m, j) is the centroid no j of sub-quantizer m.
- Parameters:
x – input vector size d
dis_table – output table, size M * ksub
-
void compute_inner_prod_table(const float *x, float *dis_table) const
-
void compute_distance_tables(size_t nx, const float *x, float *dis_tables) const
compute distance table for several vectors
- Parameters:
nx – nb of input vectors
x – input vector size nx * d
dis_table – output table, size nx * M * ksub
-
void compute_inner_prod_tables(size_t nx, const float *x, float *dis_tables) const
-
void search(const float *x, size_t nx, const uint8_t *codes, const size_t ncodes, float_maxheap_array_t *res, bool init_finalize_heap = true) const
perform a search (L2 distance)
- Parameters:
x – query vectors, size nx * d
nx – nb of queries
codes – database codes, size ncodes * code_size
ncodes – nb of nb vectors
res – heap array to store results (nh == nx)
init_finalize_heap – initialize heap (input) and sort (output)?
-
void search_ip(const float *x, size_t nx, const uint8_t *codes, const size_t ncodes, float_minheap_array_t *res, bool init_finalize_heap = true) const
same as search, but with inner product similarity
-
void compute_sdc_table()
-
void search_sdc(const uint8_t *qcodes, size_t nq, const uint8_t *bcodes, const size_t ncodes, float_maxheap_array_t *res, bool init_finalize_heap = true) const
-
void sync_transposed_centroids()
Sync transposed centroids with regular centroids. This call is needed if centroids were edited directly.
-
void clear_transposed_centroids()
Clear transposed centroids table so ones are no longer used.
Public Members
-
size_t M
number of subquantizers
-
size_t nbits
number of bits per quantization index
-
size_t dsub
dimensionality of each subvector
-
size_t ksub
number of centroids for each subquantizer
-
bool verbose
verbose during training?
-
train_type_t train_type
-
ClusteringParameters cp
parameters used during clustering
-
Index *assign_index
if non-NULL, use this index for assignment (should be of size d / M)
-
std::vector<float> centroids
Centroid table, size M * ksub * dsub. Layout: (M, ksub, dsub)
-
std::vector<float> transposed_centroids
Transposed centroid table, size M * ksub * dsub. Layout: (dsub, M, ksub)
-
std::vector<float> centroids_sq_lengths
Squared lengths of centroids, size M * ksub Layout: (M, ksub)
-
std::vector<float> sdc_table
Symmetric Distance Table.
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
enum train_type_t
-
struct PQEncoderGeneric
Public Functions
-
inline PQEncoderGeneric(uint8_t *code, int nbits, uint8_t offset = 0)
-
inline void encode(uint64_t x)
-
inline ~PQEncoderGeneric()
Public Members
-
uint8_t *code
code for this vector
-
uint8_t offset
-
const int nbits
number of bits per subquantizer index
-
uint8_t reg
-
inline PQEncoderGeneric(uint8_t *code, int nbits, uint8_t offset = 0)
-
struct PQEncoder8
Public Functions
-
inline PQEncoder8(uint8_t *code, int nbits)
-
inline void encode(uint64_t x)
Public Members
-
uint8_t *code
-
inline PQEncoder8(uint8_t *code, int nbits)
-
struct PQEncoder16
Public Functions
-
inline PQEncoder16(uint8_t *code, int nbits)
-
inline void encode(uint64_t x)
Public Members
-
uint16_t *code
-
inline PQEncoder16(uint8_t *code, int nbits)
-
struct PQDecoderGeneric
Public Functions
-
inline PQDecoderGeneric(const uint8_t *code, int nbits)
-
inline uint64_t decode()
Public Members
-
const uint8_t *code
-
uint8_t offset
-
const int nbits
-
const uint64_t mask
-
uint8_t reg
-
inline PQDecoderGeneric(const uint8_t *code, int nbits)
-
struct PQDecoder8
Public Functions
-
inline PQDecoder8(const uint8_t *code, int nbits)
-
inline uint64_t decode()
Public Members
-
const uint8_t *code
Public Static Attributes
-
static const int nbits = 8
-
inline PQDecoder8(const uint8_t *code, int nbits)
-
struct PQDecoder16
Public Functions
-
inline PQDecoder16(const uint8_t *code, int nbits)
-
inline uint64_t decode()
Public Members
-
const uint16_t *code
Public Static Attributes
-
static const int nbits = 16
-
inline PQDecoder16(const uint8_t *code, int nbits)
-
struct Quantizer
- #include <Quantizer.h>
General interface for quantizer objects
Subclassed by faiss::AdditiveQuantizer, faiss::ProductQuantizer, faiss::ScalarQuantizer
Public Functions
-
inline explicit Quantizer(size_t d = 0, size_t code_size = 0)
-
virtual void train(size_t n, const float *x) = 0
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const = 0
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
virtual void decode(const uint8_t *code, float *x, size_t n) const = 0
Decode a set of vectors
- Parameters:
codes – input codes, size n * code_size
x – output vectors, size n * d
-
inline virtual ~Quantizer()
Public Members
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
inline explicit Quantizer(size_t d = 0, size_t code_size = 0)
-
struct ResidualQuantizer : public faiss::AdditiveQuantizer
- #include <ResidualQuantizer.h>
Residual quantizer with variable number of bits per sub-quantizer
The residual centroids are stored in a big cumulative centroid table. The codes are represented either as a non-compact table of size (n, M) or as the compact output (n, code_size).
Public Types
-
using train_type_t = int
initialization
-
enum Search_type_t
Encodes how search is performed and how vectors are encoded.
Values:
-
enumerator ST_decompress
decompress database vector
-
enumerator ST_LUT_nonorm
use a LUT, don’t include norms (OK for IP or normalized vectors)
-
enumerator ST_norm_from_LUT
compute the norms from the look-up tables (cost is in O(M^2))
-
enumerator ST_norm_float
use a LUT, and store float32 norm with the vectors
-
enumerator ST_norm_qint8
use a LUT, and store 8bit-quantized norm
-
enumerator ST_norm_qint4
-
enumerator ST_norm_cqint8
use a LUT, and store non-uniform quantized norm
-
enumerator ST_norm_cqint4
-
enumerator ST_norm_lsq2x4
use a 2x4 bits lsq as norm quantizer (for fast scan)
-
enumerator ST_norm_rq2x4
use a 2x4 bits rq as norm quantizer (for fast scan)
-
enumerator ST_decompress
Public Functions
-
ResidualQuantizer(size_t d, const std::vector<size_t> &nbits, Search_type_t search_type = ST_decompress)
-
ResidualQuantizer(size_t d, size_t M, size_t nbits, Search_type_t search_type = ST_decompress)
-
ResidualQuantizer()
-
virtual void train(size_t n, const float *x) override
Train the residual quantizer.
-
void initialize_from(const ResidualQuantizer &other, int skip_M = 0)
Copy the M codebook levels from other, starting from skip_M.
-
float retrain_AQ_codebook(size_t n, const float *x)
Encode the vectors and compute codebook that minimizes the quantization error on these codes
- Parameters:
x – training vectors, size n * d
n – nb of training vectors, n >= total_codebook_size
- Returns:
returns quantization error for the new codebook with old codes
-
virtual void compute_codes_add_centroids(const float *x, uint8_t *codes, size_t n, const float *centroids = nullptr) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
centroids – centroids to be added to x, size n * d
-
void refine_beam(size_t n, size_t beam_size, const float *residuals, int new_beam_size, int32_t *new_codes, float *new_residuals = nullptr, float *new_distances = nullptr) const
lower-level encode function
- Parameters:
n – number of vectors to hanlde
residuals – vectors to encode, size (n, beam_size, d)
beam_size – input beam size
new_beam_size – output beam size (should be <= K * beam_size)
new_codes – output codes, size (n, new_beam_size, m + 1)
new_residuals – output residuals, size (n, new_beam_size, d)
new_distances – output distances, size (n, new_beam_size)
-
void refine_beam_LUT(size_t n, const float *query_norms, const float *query_cp, int new_beam_size, int32_t *new_codes, float *new_distances = nullptr) const
-
size_t memory_per_point(int beam_size = -1) const
Beam search can consume a lot of memory. This function estimates the amount of mem used by refine_beam to adjust the batch size
- Parameters:
beam_size – if != -1, override the beam size
-
void compute_codebook_tables()
Cross products used in codebook tables used for beam_LUT = 1
-
uint64_t encode_norm(float norm) const
encode a norm into norm_bits bits
-
uint32_t encode_qcint(float x) const
encode norm by non-uniform scalar quantization
-
float decode_qcint(uint32_t c) const
decode norm by non-uniform scalar quantization
-
void set_derived_values()
Train the norm quantizer.
-
void train_norm(size_t n, const float *norms)
-
inline virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Quantize a set of vectors
- Parameters:
x – input vectors, size n * d
codes – output codes, size n * code_size
-
void pack_codes(size_t n, const int32_t *codes, uint8_t *packed_codes, int64_t ld_codes = -1, const float *norms = nullptr, const float *centroids = nullptr) const
pack a series of code to bit-compact format
- Parameters:
codes – codes to be packed, size n * code_size
packed_codes – output bit-compact codes
ld_codes – leading dimension of codes
norms – norms of the vectors (size n). Will be computed if needed but not provided
centroids – centroids to be added to x, size n * d
-
virtual void decode(const uint8_t *codes, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
virtual void decode_unpacked(const int32_t *codes, float *x, size_t n, int64_t ld_codes = -1) const
Decode a set of vectors in non-packed format
- Parameters:
codes – codes to decode, size n * ld_codes
x – output vectors, size n * d
-
template<bool is_IP, Search_type_t effective_search_type>
float compute_1_distance_LUT(const uint8_t *codes, const float *LUT) const
-
virtual void compute_LUT(size_t n, const float *xq, float *LUT, float alpha = 1.0f, long ld_lut = -1) const
Compute inner-product look-up tables. Used in the centroid search functions.
- Parameters:
xq – query vector, size (n, d)
LUT – look-up table, size (n, total_codebook_size)
alpha – compute alpha * inner-product
ld_lut – leading dimension of LUT
-
void knn_centroids_inner_product(idx_t n, const float *xq, idx_t k, float *distances, idx_t *labels) const
exact IP search
-
void compute_centroid_norms(float *norms) const
For L2 search we need the L2 norms of the centroids
- Parameters:
norms – output norms table, size total_codebook_size
Public Members
-
train_type_t train_type = Train_progressive_dim
Binary or of the Train_* flags below.
-
int niter_codebook_refine = 5
number of iterations for codebook refinement.
-
int max_beam_size = 5
beam size used for training and for encoding
-
int use_beam_LUT = 0
use LUT for beam search
-
ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK
Currently used mode of approximate min-k computations. Default value is EXACT_TOPK.
-
ProgressiveDimClusteringParameters cp
clustering parameters
-
ProgressiveDimIndexFactory *assign_index_factory = nullptr
if non-NULL, use this index for assignment
-
std::vector<float> codebook_cross_products
dot products of all codebook entries with the previous codebooks size sum(codebook_offsets[m] * 2^nbits[m], m=0..M-1)
-
std::vector<float> cent_norms
norms of all codebook entries (size total_codebook_size)
-
size_t M
number of codebooks
-
size_t tot_bits = 0
total number of bits (indexes + norms)
-
size_t norm_bits = 0
bits allocated for the norms
-
size_t total_codebook_size = 0
size of the codebook in vectors
-
bool only_8bit = false
are all nbits = 8 (use faster decoder)
-
bool verbose = false
verbose during training?
-
bool is_trained = false
is trained or not
-
IndexFlat1D qnorm
store and search norms
-
size_t max_mem_distances = 5 * (size_t(1) << 30)
norms and distance matrixes with beam search can get large, so use this to control for the amount of memory that can be allocated
-
Search_type_t search_type
Also determines what’s in the codes.
-
float norm_min = NAN
min/max for quantization of norms
-
float norm_max = NAN
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
Public Static Attributes
-
static const int Train_default = 0
regular k-means (minimal amount of computation)
-
static const int Train_progressive_dim = 1
progressive dim clustering (set by default)
-
static const int Train_refine_codebook = 2
do a few iterations of codebook refinement after first level estimation
-
static const int Train_top_beam = 1024
set this bit on train_type if beam is to be trained only on the first element of the beam (faster but less accurate)
-
static const int Skip_codebook_tables = 2048
set this bit to not autmatically compute the codebook tables after training
-
using train_type_t = int
-
template<class C>
struct BlockResultHandler Subclassed by faiss::HeapBlockResultHandler< C >, faiss::RangeSearchBlockResultHandler< C >, faiss::ReservoirBlockResultHandler< C >, faiss::Top1BlockResultHandler< C >
Public Functions
-
inline explicit BlockResultHandler(size_t nq)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2)
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline virtual void end_multiple()
-
inline virtual ~BlockResultHandler()
Public Members
-
size_t nq
-
size_t i0 = 0
-
size_t i1 = 0
-
inline explicit BlockResultHandler(size_t nq)
-
template<class C>
struct Top1BlockResultHandler : public faiss::BlockResultHandler<C> -
Public Functions
-
inline virtual void begin_multiple(size_t i0, size_t i1) final
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab_2) final
add results for query i0..i1 and j0..j1
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline virtual void end_multiple()
-
struct SingleResultHandler : public faiss::ResultHandler<C>
Public Functions
-
inline explicit SingleResultHandler(Top1BlockResultHandler &hr)
-
inline void begin(const size_t current_idx_2)
begin results for query # i
-
inline void end()
series of results for query i is done
-
inline explicit SingleResultHandler(Top1BlockResultHandler &hr)
-
inline virtual void begin_multiple(size_t i0, size_t i1) final
-
template<class C>
struct HeapBlockResultHandler : public faiss::BlockResultHandler<C> -
Public Functions
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2) final
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab) final
add results for query i0..i1 and j0..j1
-
inline virtual void end_multiple() final
series of results for queries i0..i1 is done
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
struct SingleResultHandler : public faiss::ResultHandler<C>
Public Functions
-
inline explicit SingleResultHandler(HeapBlockResultHandler &hr)
-
inline void begin(size_t i)
begin results for query # i
-
inline void end()
series of results for query i is done
-
inline explicit SingleResultHandler(HeapBlockResultHandler &hr)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2) final
-
template<class C>
struct ReservoirTopN : public faiss::ResultHandler<C> - #include <ResultHandler.h>
Reservoir for a single query.
Subclassed by faiss::ReservoirBlockResultHandler< C >::SingleResultHandler
Public Functions
-
inline ReservoirTopN()
-
inline void shrink_fuzzy()
-
inline void shrink()
-
inline ReservoirTopN()
-
template<class C>
struct ReservoirBlockResultHandler : public faiss::BlockResultHandler<C> -
Public Functions
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2)
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab)
add results for query i0..i1 and j0..j1
-
inline virtual void end_multiple() final
series of results for queries i0..i1 is done
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
Public Members
-
T *heap_dis_tab
-
TI *heap_ids_tab
-
int64_t k
-
size_t capacity
-
std::vector<ReservoirTopN<C>> reservoirs
-
size_t nq
-
size_t i0 = 0
-
size_t i1 = 0
-
struct SingleResultHandler : public faiss::ReservoirTopN<C>
-
Public Functions
-
inline explicit SingleResultHandler(ReservoirBlockResultHandler &hr)
-
inline void begin(size_t qno_2)
begin results for query # i
-
inline void end()
series of results for query qno is done
-
inline void shrink_fuzzy()
-
inline void shrink()
-
inline explicit SingleResultHandler(ReservoirBlockResultHandler &hr)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2)
-
template<class C>
struct RangeSearchBlockResultHandler : public faiss::BlockResultHandler<C> -
Public Functions
-
inline RangeSearchBlockResultHandler(RangeSearchResult *res, float radius)
-
inline virtual void begin_multiple(size_t i0_2, size_t i1_2)
begin
-
inline void add_results(size_t j0, size_t j1, const T *dis_tab)
add results for query i0..i1 and j0..j1
-
inline ~RangeSearchBlockResultHandler()
-
inline virtual void add_results(size_t, size_t, const typename C::T*)
-
inline virtual void end_multiple()
Public Members
-
RangeSearchResult *res
-
T radius
-
std::vector<RangeSearchPartialResult*> partial_results
-
std::vector<size_t> j0s
-
int pr = 0
-
size_t nq
-
size_t i0 = 0
-
size_t i1 = 0
-
struct SingleResultHandler : public faiss::ResultHandler<C>
Public Functions
-
inline explicit SingleResultHandler(RangeSearchBlockResultHandler &rh)
-
inline void begin(size_t i)
begin results for query # i
-
inline void end()
series of results for query i is done
-
inline ~SingleResultHandler()
-
inline explicit SingleResultHandler(RangeSearchBlockResultHandler &rh)
-
inline RangeSearchBlockResultHandler(RangeSearchResult *res, float radius)
-
struct ScalarQuantizer : public faiss::Quantizer
- #include <ScalarQuantizer.h>
The uniform quantizer has a range [vmin, vmax]. The range can be the same for all dimensions (uniform) or specific per dimension (default).
Public Types
-
enum QuantizerType
Values:
-
enumerator QT_8bit
8 bits per component
-
enumerator QT_4bit
4 bits per component
-
enumerator QT_8bit_uniform
same, shared range for all dimensions
-
enumerator QT_4bit_uniform
-
enumerator QT_fp16
-
enumerator QT_8bit_direct
fast indexing of uint8s
-
enumerator QT_6bit
6 bits per component
-
enumerator QT_8bit
-
enum RangeStat
The uniform encoder can estimate the range of representable values of the unform encoder using different statistics. Here rs = rangestat_arg
Values:
-
enumerator RS_minmax
[min - rs*(max-min), max + rs*(max-min)]
-
enumerator RS_meanstd
[mean - std * rs, mean + std * rs]
-
enumerator RS_quantiles
[Q(rs), Q(1-rs)]
-
enumerator RS_optim
alternate optimization of reconstruction error
-
enumerator RS_minmax
Public Functions
-
ScalarQuantizer(size_t d, QuantizerType qtype)
-
ScalarQuantizer()
-
void set_derived_sizes()
updates internal values based on qtype and d
-
virtual void train(size_t n, const float *x) override
Train the quantizer
- Parameters:
x – training vectors, size n * d
-
virtual void compute_codes(const float *x, uint8_t *codes, size_t n) const override
Encode a set of vectors
- Parameters:
x – vectors to encode, size n * d
codes – output codes, size n * code_size
-
virtual void decode(const uint8_t *code, float *x, size_t n) const override
Decode a set of vectors
- Parameters:
codes – codes to decode, size n * code_size
x – output vectors, size n * d
-
SQuantizer *select_quantizer() const
-
SQDistanceComputer *get_distance_computer(MetricType metric = METRIC_L2) const
-
InvertedListScanner *select_InvertedListScanner(MetricType mt, const Index *quantizer, bool store_pairs, const IDSelector *sel, bool by_residual = false) const
Public Members
-
QuantizerType qtype = QT_8bit
-
float rangestat_arg = 0
-
size_t bits = 0
bits per scalar code
-
std::vector<float> trained
trained values (including the range)
-
size_t d
size of the input vectors
-
size_t code_size
bytes per indexed vector
-
struct SQDistanceComputer : public faiss::FlatCodesDistanceComputer
Public Functions
-
inline SQDistanceComputer()
-
virtual float query_to_code(const uint8_t *code) const = 0
-
inline virtual float distance_to_code(const uint8_t *code) final
compute distance of current query to an encoded vector
-
virtual void set_query(const float *x) = 0
called before computing distances. Pointer x should remain valid while operator () is called
-
inline SQDistanceComputer()
-
struct SQuantizer
Public Functions
-
virtual void encode_vector(const float *x, uint8_t *code) const = 0
-
virtual void decode_vector(const uint8_t *code, float *x) const = 0
-
inline virtual ~SQuantizer()
-
virtual void encode_vector(const float *x, uint8_t *code) const = 0
-
enum QuantizerType
-
struct SIMDResultHandler
Subclassed by faiss::SIMDResultHandlerToFloat, faiss::simd_result_handlers::DummyResultHandler, faiss::simd_result_handlers::FixedStorageHandler< NQ, BB >, faiss::simd_result_handlers::StoreResultHandler
Public Functions
-
virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
virtual void set_block_origin(size_t i0, size_t j0) = 0
set the sub-matrix that is being computed
-
inline virtual ~SIMDResultHandler()
Public Members
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0
-
struct SIMDResultHandlerToFloat : public faiss::SIMDResultHandler
Subclassed by faiss::simd_result_handlers::ResultHandlerCompare< C, false >, faiss::simd_result_handlers::ResultHandlerCompare< C, with_id_map >
Public Functions
-
inline SIMDResultHandlerToFloat(size_t nq, size_t ntotal)
-
inline virtual void begin(const float *norms)
-
inline virtual void end()
-
virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
virtual void set_block_origin(size_t i0, size_t j0) = 0
set the sub-matrix that is being computed
Public Members
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
inline SIMDResultHandlerToFloat(size_t nq, size_t ntotal)
-
template<typename IndexT>
class ThreadedIndex : public IndexT - #include <ThreadedIndex.h>
A holder of indices in a collection of threads The interface to this class itself is not thread safe
Subclassed by faiss::IndexReplicasTemplate< IndexT >, faiss::IndexShardsTemplate< IndexT >
Public Functions
-
explicit ThreadedIndex(bool threaded)
-
explicit ThreadedIndex(int d, bool threaded)
-
~ThreadedIndex() override
-
virtual void addIndex(IndexT *index)
override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.
-
void removeIndex(IndexT *index)
Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.
-
void runOnIndex(std::function<void(int, IndexT*)> f)
Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)
-
void reset() override
faiss::Index API All indices receive the same call
-
inline int count() const
Returns the number of sub-indices.
-
inline IndexT *at(size_t i)
Returns the i-th sub-index.
-
inline const IndexT *at(size_t i) const
Returns the i-th sub-index (const version)
Public Members
-
bool own_indices = false
Whether or not we are responsible for deleting our contained indices.
Protected Functions
-
virtual void onAfterAddIndex(IndexT *index)
Called just after an index is added.
-
virtual void onAfterRemoveIndex(IndexT *index)
Called just after an index is removed.
-
explicit ThreadedIndex(bool threaded)
-
struct SearchParameters
- #include <Index.h>
Parent class for the optional search paramenters.
Sub-classes with additional search parameters should inherit this class. Ownership of the object fields is always to the caller.
Subclassed by faiss::IndexRefineSearchParameters, faiss::SearchParametersHNSW, faiss::SearchParametersIVF, faiss::SearchParametersPQ, faiss::SearchParametersPreTransform, faiss::SearchParametersResidualCoarseQuantizer
Public Functions
-
inline virtual ~SearchParameters()
make sure we can dynamic_cast this
Public Members
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline virtual ~SearchParameters()
-
struct Index
- #include <Index.h>
Abstract structure for an index, supports adding vectors and searching them.
All vectors provided at add or search time are 32-bit float arrays, although the internal representation may vary.
Subclassed by faiss::AdditiveCoarseQuantizer, faiss::IndexFastScan, faiss::IndexFlatCodes, faiss::IndexHNSW, faiss::IndexIVF, faiss::IndexIVFIndependentQuantizer, faiss::IndexLattice, faiss::IndexNNDescent, faiss::IndexNSG, faiss::IndexPreTransform, faiss::IndexRandom, faiss::IndexRefine, faiss::IndexRowwiseMinMaxBase, faiss::IndexSplitVectors, faiss::MultiIndexQuantizer, faiss::gpu::GpuIndex
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
inline explicit Index(idx_t d = 0, MetricType metric = METRIC_L2)
-
virtual ~Index()
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) = 0
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const = 0
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reset() = 0
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using component_t = float
-
struct Index2Layer : public faiss::IndexFlatCodes
- #include <Index2Layer.h>
Same as an IndexIVFPQ without the inverted lists: codes are stored sequentially
The class is mainly inteded to store encoded vectors that can be accessed randomly, the search function is not implemented.
Public Functions
-
Index2Layer(Index *quantizer, size_t nlist, int M, int nbit = 8, MetricType metric = METRIC_L2)
-
Index2Layer()
-
~Index2Layer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
not implemented
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void transfer_to_IVFPQ(IndexIVFPQ &other) const
transfer the flat codes to an IVFPQ index
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const
a FlatCodesDistanceComputer offers a distance_to_code method
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
Level1Quantizer q1
first level quantizer
-
ProductQuantizer pq
second level quantizer is always a PQ
-
size_t code_size_1
size of the code for the first level (ceil(log8(q1.nlist)))
-
size_t code_size_2
size of the code for the second level
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
Index2Layer(Index *quantizer, size_t nlist, int M, int nbit = 8, MetricType metric = METRIC_L2)
-
struct IndexAdditiveQuantizer : public faiss::IndexFlatCodes
- #include <IndexAdditiveQuantizer.h>
Abstract class for additive quantizers. The search functions are in common.
Subclassed by faiss::IndexLocalSearchQuantizer, faiss::IndexProductLocalSearchQuantizer, faiss::IndexProductResidualQuantizer, faiss::IndexResidualQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
explicit IndexAdditiveQuantizer(idx_t d, AdditiveQuantizer *aq, MetricType metric = METRIC_L2)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexResidualQuantizer : public faiss::IndexAdditiveQuantizer
- #include <IndexAdditiveQuantizer.h>
Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexResidualQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexResidualQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
-
IndexResidualQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexLocalSearchQuantizer : public faiss::IndexAdditiveQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexLocalSearchQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexLocalSearchQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductResidualQuantizer : public faiss::IndexAdditiveQuantizer
- #include <IndexAdditiveQuantizer.h>
Index based on a product residual quantizer.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductResidualQuantizer(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
-
IndexProductResidualQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ProductResidualQuantizer prq
The product residual quantizer used to encode the vectors.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductLocalSearchQuantizer : public faiss::IndexAdditiveQuantizer
- #include <IndexAdditiveQuantizer.h>
Index based on a product local search quantizer.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductLocalSearchQuantizer(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
-
IndexProductLocalSearchQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ProductLocalSearchQuantizer plsq
The product local search quantizer used to encode the vectors.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct AdditiveCoarseQuantizer : public faiss::Index
- #include <IndexAdditiveQuantizer.h>
A “virtual” index where the elements are the residual quantizer centroids.
Intended for use as a coarse quantizer in an IndexIVF.
Subclassed by faiss::LocalSearchCoarseQuantizer, faiss::ResidualCoarseQuantizer
Public Functions
-
explicit AdditiveCoarseQuantizer(idx_t d = 0, AdditiveQuantizer *aq = nullptr, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
explicit AdditiveCoarseQuantizer(idx_t d = 0, AdditiveQuantizer *aq = nullptr, MetricType metric = METRIC_L2)
-
struct SearchParametersResidualCoarseQuantizer : public faiss::SearchParameters
Public Functions
-
inline ~SearchParametersResidualCoarseQuantizer()
Public Members
-
float beam_factor = 4.0f
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline ~SearchParametersResidualCoarseQuantizer()
-
struct ResidualCoarseQuantizer : public faiss::AdditiveCoarseQuantizer
- #include <IndexAdditiveQuantizer.h>
The ResidualCoarseQuantizer is a bit specialized compared to the default AdditiveCoarseQuantizer because it can use a beam search at search time (slow but may be useful for very large vocabularies)
Public Functions
-
void set_beam_factor(float new_beam_factor)
computes centroid norms if required
-
ResidualCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
ResidualCoarseQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
void initialize_from(const ResidualCoarseQuantizer &other)
Copy the M first codebook levels from other. Useful to crop a ResidualQuantizer to its first M quantizers.
-
ResidualCoarseQuantizer()
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
float beam_factor = 4.0f
factor between the beam size and the search k if negative, use exact search-to-centroid
-
std::vector<float> centroid_norms
norms of centroids, useful for knn-search
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
void set_beam_factor(float new_beam_factor)
-
struct LocalSearchCoarseQuantizer : public faiss::AdditiveCoarseQuantizer
-
Public Functions
-
LocalSearchCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
LocalSearchCoarseQuantizer()
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
LocalSearchQuantizer lsq
The residual quantizer used to encode the vectors.
-
std::vector<float> centroid_norms
norms of centroids, useful for knn-search
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
LocalSearchCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
-
struct IndexAdditiveQuantizerFastScan : public faiss::IndexFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Fast scan version of IndexAQ. Works for 4-bit AQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Subclassed by faiss::IndexLocalSearchQuantizerFastScan, faiss::IndexProductLocalSearchQuantizerFastScan, faiss::IndexProductResidualQuantizerFastScan, faiss::IndexResidualQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
explicit IndexAdditiveQuantizerFastScan(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
IndexAdditiveQuantizerFastScan()
-
~IndexAdditiveQuantizerFastScan() override
-
explicit IndexAdditiveQuantizerFastScan(const IndexAdditiveQuantizer &orig, int bbs = 32)
build from an existing IndexAQ
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexResidualQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexResidualQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexLocalSearchQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a local search quantizer. Stored vectors are approximated by local search quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexLocalSearchQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductResidualQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a product residual quantizer. Stored vectors are approximated by product residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductResidualQuantizerFastScan(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexProductResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
ProductResidualQuantizer prq
The product residual quantizer used to encode the vectors.
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexProductLocalSearchQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a product local search quantizer. Stored vectors are approximated by product local search quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexProductLocalSearchQuantizerFastScan(int d, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexProductLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
ProductLocalSearchQuantizer plsq
The product local search quantizer used to encode the vectors.
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexBinary
- #include <IndexBinary.h>
Abstract structure for a binary index.
Supports adding vertices and searching them.
All queries are symmetric because there is no distinction between codes and vectors.
Subclassed by faiss::IndexBinaryFlat, faiss::IndexBinaryFromFloat, faiss::IndexBinaryHNSW, faiss::IndexBinaryHash, faiss::IndexBinaryIVF, faiss::IndexBinaryMultiHash, faiss::gpu::GpuIndexBinaryFlat
Public Types
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
explicit IndexBinary(idx_t d = 0, MetricType metric = METRIC_L2)
-
virtual ~IndexBinary()
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add(idx_t n, const uint8_t *x) = 0
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const = 0
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual void reset() = 0
Removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
idx_t ntotal = 0
total nb of indexed vectors
-
bool verbose = false
verbosity level
-
bool is_trained = true
set if the Index does not require training, or if training is done already
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
using component_t = uint8_t
-
struct IndexBinaryFlat : public faiss::IndexBinary
- #include <IndexBinaryFlat.h>
Index that stores the full vectors and performs exhaustive search.
Public Functions
-
explicit IndexBinaryFlat(idx_t d)
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual size_t remove_ids(const IDSelector &sel) override
Remove some ids. Note that because of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted.
-
inline IndexBinaryFlat()
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
std::vector<uint8_t> xb
database vectors, size ntotal * d / 8
-
bool use_heap = true
Select between using a heap or counting to select the k smallest values when scanning inverted lists.
-
size_t query_batch_size = 32
-
ApproxTopK_mode_t approx_topk_mode = ApproxTopK_mode_t::EXACT_TOPK
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
explicit IndexBinaryFlat(idx_t d)
-
struct IndexBinaryFromFloat : public faiss::IndexBinary
- #include <IndexBinaryFromFloat.h>
IndexBinary backed by a float Index.
Supports adding vertices and searching them.
All queries are symmetric because there is no distinction between codes and vectors.
Public Functions
-
IndexBinaryFromFloat()
-
explicit IndexBinaryFromFloat(Index *index)
-
~IndexBinaryFromFloat()
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void train(idx_t n, const uint8_t *x) override
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
Index *index = nullptr
-
bool own_fields = false
Whether object owns the index pointer.
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
IndexBinaryFromFloat()
-
struct IndexBinaryHash : public faiss::IndexBinary
- #include <IndexBinaryHash.h>
just uses the b first bits as a hash value
Public Types
-
using InvertedListMap = std::unordered_map<idx_t, InvertedList>
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
IndexBinaryHash(int d, int b)
-
IndexBinaryHash()
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
void display() const
-
size_t hashtable_size() const
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
InvertedListMap invlists
-
int b
-
int nflip
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
struct InvertedList
Public Functions
-
void add(idx_t id, size_t code_size, const uint8_t *code)
-
void add(idx_t id, size_t code_size, const uint8_t *code)
-
using InvertedListMap = std::unordered_map<idx_t, InvertedList>
-
struct IndexBinaryHashStats
Public Functions
-
inline IndexBinaryHashStats()
-
void reset()
Public Members
-
size_t nq
-
size_t n0
-
size_t nlist
-
size_t ndis
-
inline IndexBinaryHashStats()
-
struct IndexBinaryMultiHash : public faiss::IndexBinary
- #include <IndexBinaryHash.h>
just uses the b first bits as a hash value
Public Functions
-
IndexBinaryMultiHash(int d, int nhash, int b)
-
IndexBinaryMultiHash()
-
~IndexBinaryMultiHash()
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
size_t hashtable_size() const
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
IndexBinaryFlat *storage
-
bool own_fields
-
int nhash
nb of hash maps
-
int b
nb bits per hash map
-
int nflip
nb bit flips to use at search time
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
IndexBinaryMultiHash(int d, int nhash, int b)
-
struct IndexBinaryHNSW : public faiss::IndexBinary
- #include <IndexBinaryHNSW.h>
The HNSW index is a normal random-access index with a HNSW link structure built on top
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
explicit IndexBinaryHNSW()
-
explicit IndexBinaryHNSW(int d, int M = 32)
-
explicit IndexBinaryHNSW(IndexBinary *storage, int M = 32)
-
~IndexBinaryHNSW() override
-
DistanceComputer *get_distance_computer() const
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void train(idx_t n, const uint8_t *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
HNSW hnsw
-
bool own_fields
-
IndexBinary *storage
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexBinaryIVF : public faiss::IndexBinary
- #include <IndexBinaryIVF.h>
Index based on a inverted file (IVF)
In the inverted file, the quantizer (an IndexBinary instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.
Otherwise the object is similar to the IndexIVF
Public Functions
-
IndexBinaryIVF(IndexBinary *quantizer, size_t d, size_t nlist)
The Inverted file takes a quantizer (an IndexBinary) on input, which implements the function mapping a vector to a list identifier. The pointer is borrowed: the quantizer should not be deleted while the IndexBinaryIVF is in use.
-
IndexBinaryIVF()
-
~IndexBinaryIVF() override
-
virtual void reset() override
Removes all elements from the database.
-
virtual void train(idx_t n, const uint8_t *x) override
Trains the quantizer.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void add_core(idx_t n, const uint8_t *x, const idx_t *xids, const idx_t *precomputed_idx)
Implementation of vector addition where the vector assignments are predefined.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void search_preassigned(idx_t n, const uint8_t *x, idx_t k, const idx_t *assign, const int32_t *centroid_dis, int32_t *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr) const
Search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. search() calls this.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
-
virtual BinaryInvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void range_search_preassigned(idx_t n, const uint8_t *x, int radius, const idx_t *assign, const int32_t *centroid_dis, RangeSearchResult *result) const
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d / 8
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d / 8)
-
virtual void reconstruct_from_offset(idx_t list_no, idx_t offset, uint8_t *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void merge_from(IndexBinary &other, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
inline size_t get_list_size(size_t list_no) const
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
void display() const
Display the actual class name and some more info.
Public Members
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = true
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
bool use_heap = true
Select between using a heap or counting to select the k smallest values when scanning inverted lists.
-
bool per_invlist_search = false
collect computations per batch
-
DirectMap direct_map
map for direct access to the elements. Enables reconstruct().
-
IndexBinary *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of possible key values
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
bool verbose = false
verbosity level
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
-
IndexBinaryIVF(IndexBinary *quantizer, size_t d, size_t nlist)
-
struct BinaryInvertedListScanner
Public Functions
-
virtual void set_query(const uint8_t *query_vector) = 0
from now on we handle this query.
-
virtual void set_list(idx_t list_no, uint8_t coarse_dis) = 0
following codes come from this inverted list
-
virtual uint32_t distance_to_code(const uint8_t *code) const = 0
compute a single query-to-code distance
-
virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, int32_t *distances, idx_t *labels, size_t k) const = 0
compute the distances to codes. (distances, labels) should be organized as a min- or max-heap
- Parameters:
n – number of codes to scan
codes – codes to scan (n * code_size)
ids – corresponding ids (ignored if store_pairs)
distances – heap distances (size k)
labels – heap labels (size k)
k – heap size
-
virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, int radius, RangeQueryResult &result) const = 0
-
inline virtual ~BinaryInvertedListScanner()
-
virtual void set_query(const uint8_t *query_vector) = 0
-
struct IndexFastScan : public faiss::Index
- #include <IndexFastScan.h>
Fast scan version of IndexPQ and IndexAQ. Works for 4-bit PQ and AQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions. The trailing codes (padding codes that are added to complete the last code) are garbage.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Subclassed by faiss::IndexAdditiveQuantizerFastScan, faiss::IndexPQFastScan
Public Functions
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
IndexFastScan()
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const = 0
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const = 0
-
void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
struct FastScanStats
Public Functions
-
inline FastScanStats()
-
inline void reset()
Public Members
-
uint64_t t0
-
uint64_t t1
-
uint64_t t2
-
uint64_t t3
-
inline FastScanStats()
-
struct IndexFlat : public faiss::IndexFlatCodes
- #include <IndexFlat.h>
Index that stores the full vectors and performs exhaustive search
Subclassed by faiss::IndexFlatIP, faiss::IndexFlatL2
Public Functions
-
explicit IndexFlat(idx_t d, MetricType metric = METRIC_L2)
- Parameters:
d – dimensionality of the input vectors
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
inline IndexFlat()
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
explicit IndexFlat(idx_t d, MetricType metric = METRIC_L2)
-
struct IndexFlatIP : public faiss::IndexFlat
-
Public Functions
-
inline explicit IndexFlatIP(idx_t d)
-
inline IndexFlatIP()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
inline explicit IndexFlatIP(idx_t d)
-
struct IndexFlatL2 : public faiss::IndexFlat
Subclassed by faiss::IndexFlat1D
Public Functions
-
inline explicit IndexFlatL2(idx_t d)
- Parameters:
d – dimensionality of the input vectors
-
inline IndexFlatL2()
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
void sync_l2norms()
-
void clear_l2norms()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
inline explicit IndexFlatL2(idx_t d)
-
struct IndexFlat1D : public faiss::IndexFlatL2
- #include <IndexFlat.h>
optimized version for 1D “vectors”.
Public Functions
-
explicit IndexFlat1D(bool continuous_update = true)
-
void update_permutation()
if not continuous_update, call this between the last add and the first search
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
Warn: the distances returned are L1 not L2.
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
void sync_l2norms()
-
void clear_l2norms()
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
bool continuous_update = true
is the permutation updated continuously?
-
std::vector<float> cached_l2norms
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
explicit IndexFlat1D(bool continuous_update = true)
-
struct IndexFlatCodes : public faiss::Index
- #include <IndexFlatCodes.h>
Index that encodes all vectors as fixed-size codes (size code_size). Storage is in the codes vector
Subclassed by faiss::Index2Layer, faiss::IndexAdditiveQuantizer, faiss::IndexFlat, faiss::IndexLSH, faiss::IndexPQ, faiss::IndexScalarQuantizer
Public Functions
-
IndexFlatCodes()
-
IndexFlatCodes(size_t code_size, idx_t d, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
default add uses sa_encode
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const
a FlatCodesDistanceComputer offers a distance_to_code method
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
void permute_entries(const idx_t *perm)
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const = 0
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
IndexFlatCodes()
-
struct IndexHNSW : public faiss::Index
- #include <IndexHNSW.h>
The HNSW index is a normal random-access index with a HNSW link structure built on top
Subclassed by faiss::IndexHNSW2Level, faiss::IndexHNSWFlat, faiss::IndexHNSWPQ, faiss::IndexHNSWSQ
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
explicit IndexHNSW(int d = 0, int M = 32, MetricType metric = METRIC_L2)
-
explicit IndexHNSW(Index *storage, int M = 32)
-
~IndexHNSW() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWFlat : public faiss::IndexHNSW
- #include <IndexHNSW.h>
Flat index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSWFlat()
-
IndexHNSWFlat(int d, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWPQ : public faiss::IndexHNSW
- #include <IndexHNSW.h>
PQ index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSWPQ()
-
IndexHNSWPQ(int d, int pq_m, int M, int pq_nbits = 8)
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWSQ : public faiss::IndexHNSW
- #include <IndexHNSW.h>
SQ index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSWSQ()
-
IndexHNSWSQ(int d, ScalarQuantizer::QuantizerType qtype, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSW2Level : public faiss::IndexHNSW
- #include <IndexHNSW.h>
2-level code structure with fast random access
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexHNSW2Level()
-
IndexHNSW2Level(Index *quantizer, size_t nlist, int m_pq, int M)
-
void flip_to_ivf()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
void permute_entries(const idx_t *perm)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
typedef HNSW::storage_idx_t storage_idx_t
-
template<typename IndexT>
struct IndexIDMapTemplate : public IndexT - #include <IndexIDMap.h>
Index that translates search results to ids
Subclassed by faiss::IndexIDMap2Template< IndexT >
Public Types
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexIDMapTemplate(IndexT *index)
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void add(idx_t n, const component_t *x) override
this will fail. Use add_with_ids
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
-
void train(idx_t n, const component_t *x) override
-
void reset() override
-
size_t remove_ids(const IDSelector &sel) override
remove ids adapted to IndexFlat
-
void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
-
void check_compatible_for_merge(const IndexT &otherIndex) const override
-
~IndexIDMapTemplate() override
-
inline IndexIDMapTemplate()
-
using component_t = typename IndexT::component_t
-
template<typename IndexT>
struct IndexIDMap2Template : public faiss::IndexIDMapTemplate<IndexT> - #include <IndexIDMap.h>
same as IndexIDMap but also provides an efficient reconstruction implementation via a 2-way index
Public Types
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexIDMap2Template(IndexT *index)
-
void construct_rev_map()
make the rev_map from scratch
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
-
size_t remove_ids(const IDSelector &sel) override
-
void reconstruct(idx_t key, component_t *recons) const override
-
void check_consistency() const
check that the rev_map and the id_map are in sync
-
inline ~IndexIDMap2Template() override
-
inline IndexIDMap2Template()
-
void add(idx_t n, const component_t *x) override
this will fail. Use add_with_ids
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
-
void train(idx_t n, const component_t *x) override
-
void reset() override
-
void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
-
void check_compatible_for_merge(const IndexT &otherIndex) const override
-
using component_t = typename IndexT::component_t
-
struct IDSelectorTranslated : public faiss::IDSelector
Public Functions
-
inline IDSelectorTranslated(const std::vector<int64_t> &id_map, const IDSelector *sel)
-
inline IDSelectorTranslated(IndexBinaryIDMap &index_idmap, const IDSelector *sel)
-
inline IDSelectorTranslated(IndexIDMap &index_idmap, const IDSelector *sel)
-
inline virtual bool is_member(idx_t id) const override
-
inline IDSelectorTranslated(const std::vector<int64_t> &id_map, const IDSelector *sel)
-
struct Level1Quantizer
- #include <IndexIVF.h>
Encapsulates a quantizer object for the IndexIVF
The class isolates the fields that are independent of the storage of the lists (especially training)
Subclassed by faiss::IndexIVFInterface, faiss::IndexShardsIVF
Public Functions
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
-
Level1Quantizer(Index *quantizer, size_t nlist)
-
Level1Quantizer()
-
~Level1Quantizer()
Public Members
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
-
struct SearchParametersIVF : public faiss::SearchParameters
Subclassed by faiss::IVFPQSearchParameters
Public Functions
-
inline virtual ~SearchParametersIVF()
Public Members
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
SearchParameters *quantizer_params = nullptr
-
void *inverted_list_context = nullptr
context object to pass to InvertedLists
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline virtual ~SearchParametersIVF()
-
struct IndexIVFInterface : public faiss::Level1Quantizer
Subclassed by faiss::IndexIVF, faiss::gpu::GpuIndexIVF
Public Functions
-
inline explicit IndexIVFInterface(Index *quantizer = nullptr, size_t nlist = 0)
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const = 0
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const = 0
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
inline virtual ~IndexIVFInterface()
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
inline explicit IndexIVFInterface(Index *quantizer = nullptr, size_t nlist = 0)
-
struct IndexIVF : public faiss::Index, public faiss::IndexIVFInterface
- #include <IndexIVF.h>
Index based on a inverted file (IVF)
In the inverted file, the quantizer (an Index instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.
The inverted list object is required only after trainng. If none is set externally, an ArrayInvertedLists is used automatically.
At search time, the vector to be searched is also quantized, and only the list corresponding to the quantization index is searched. This speeds up the search by making it non-exhaustive. This can be relaxed using multi-probe search: a few (nprobe) quantization indices are selected and several inverted lists are visited.
Sub-classes implement a post-filtering of the index that refines the distance estimation from the query to databse vectors.
Subclassed by faiss::IndexIVFAdditiveQuantizer, faiss::IndexIVFFastScan, faiss::IndexIVFFlat, faiss::IndexIVFPQ, faiss::IndexIVFScalarQuantizer, faiss::IndexIVFSpectralHash
Public Functions
-
IndexIVF(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
The Inverted file takes a quantizer (an Index) on input, which implements the function mapping a vector to a list identifier.
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add(idx_t n, const float *x) override
Calls add_with_ids with NULL ids.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const = 0
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign)
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
~IndexIVF() override
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
IndexIVF()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
-
IndexIVF(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
-
struct InvertedListScanner
- #include <IndexIVF.h>
Object that handles a query. The inverted lists to scan are provided externally. The object has a lot of state, but distance_to_code and scan_codes can be called in multiple threads
Public Functions
-
inline InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr)
-
virtual void set_query(const float *query_vector) = 0
from now on we handle this query.
-
virtual void set_list(idx_t list_no, float coarse_dis) = 0
following codes come from this inverted list
-
virtual float distance_to_code(const uint8_t *code) const = 0
compute a single query-to-code distance
-
virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, float *distances, idx_t *labels, size_t k) const
scan a set of codes, compute distances to current query and update heap of results if necessary. Default implemetation calls distance_to_code.
- Parameters:
n – number of codes to scan
codes – codes to scan (n * code_size)
ids – corresponding ids (ignored if store_pairs)
distances – heap distances (size k)
labels – heap labels (size k)
k – heap size
- Returns:
number of heap updates performed
-
virtual size_t iterate_codes(InvertedListsIterator *iterator, float *distances, idx_t *labels, size_t k, size_t &list_size) const
-
virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, float radius, RangeQueryResult &result) const
scan a set of codes, compute distances to current query and update results if distances are below radius
(default implementation fails)
-
virtual void iterate_codes_range(InvertedListsIterator *iterator, float radius, RangeQueryResult &result, size_t &list_size) const
-
inline virtual ~InvertedListScanner()
Public Members
-
idx_t list_no = -1
remember current list
-
bool keep_max = false
keep maximum instead of minimum
-
bool store_pairs
store positions in invlists rather than labels
-
const IDSelector *sel
search in this subset of ids
-
size_t code_size = 0
used in default implementation of scan_codes
-
inline InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr)
-
struct IndexIVFStats
-
Public Members
-
size_t nq
-
size_t nlist
-
size_t ndis
-
size_t nheap_updates
-
double quantization_time
-
double search_time
-
size_t nq
-
struct IndexIVFAdditiveQuantizer : public faiss::IndexIVF
- #include <IndexIVFAdditiveQuantizer.h>
Abstract class for IVF additive quantizers. The search functions are in common.
Subclassed by faiss::IndexIVFLocalSearchQuantizer, faiss::IndexIVFProductLocalSearchQuantizer, faiss::IndexIVFProductResidualQuantizer, faiss::IndexIVFResidualQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq, Index *quantizer, size_t d, size_t nlist, MetricType metric = METRIC_L2)
-
explicit IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
~IndexIVFAdditiveQuantizer() override
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
int use_precomputed_table = 0
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFResidualQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
-
IndexIVFResidualQuantizer()
-
virtual ~IndexIVFResidualQuantizer()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
int use_precomputed_table = 0
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFLocalSearchQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFLocalSearchQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexIVFLocalSearchQuantizer()
-
virtual ~IndexIVFLocalSearchQuantizer()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
LocalSearchQuantizer lsq
The LSQ quantizer used to encode the vectors.
-
int use_precomputed_table = 0
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFProductResidualQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a product residual quantizer. Stored vectors are approximated by product residual quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFProductResidualQuantizer(Index *quantizer, size_t d, size_t nlist, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of residual quantizers
Msub – number of subquantizers per RQ
nbits – number of bit per subvector index
-
IndexIVFProductResidualQuantizer()
-
virtual ~IndexIVFProductResidualQuantizer()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
ProductResidualQuantizer prq
The product residual quantizer used to encode the vectors.
-
int use_precomputed_table = 0
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFProductLocalSearchQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a product local search quantizer. Stored vectors are approximated by product local search quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFProductLocalSearchQuantizer(Index *quantizer, size_t d, size_t nlist, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
nsplits – number of local search quantizers
Msub – number of subquantizers per LSQ
nbits – number of bit per subvector index
-
IndexIVFProductLocalSearchQuantizer()
-
virtual ~IndexIVFProductLocalSearchQuantizer()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
ProductLocalSearchQuantizer plsq
The product local search quantizer used to encode the vectors.
-
int use_precomputed_table = 0
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFAdditiveQuantizerFastScan : public faiss::IndexIVFFastScan
- #include <IndexIVFAdditiveQuantizerFastScan.h>
Fast scan version of IVFAQ. Works for 4-bit AQ for now.
The codes in the inverted lists are not stored sequentially but grouped in blocks of size bbs. This makes it possible to very quickly compute distances with SIMD instructions.
Implementations (implem): 0: auto-select implementation (default) 1: orig’s search, re-implemented 2: orig’s search, re-ordered by invlist 10: optimizer int16 search, collect results in heap, no qbs 11: idem, collect results in reservoir 12: optimizer int16 search, collect results in heap, uses qbs 13: idem, collect results in reservoir
Subclassed by faiss::IndexIVFLocalSearchQuantizerFastScan, faiss::IndexIVFProductLocalSearchQuantizerFastScan, faiss::IndexIVFProductResidualQuantizerFastScan, faiss::IndexIVFResidualQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFAdditiveQuantizerFastScan(Index *quantizer, AdditiveQuantizer *aq, size_t d, size_t nlist, MetricType metric = METRIC_L2, int bbs = 32)
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
IndexIVFAdditiveQuantizerFastScan()
-
~IndexIVFAdditiveQuantizerFastScan() override
-
explicit IndexIVFAdditiveQuantizerFastScan(const IndexIVFAdditiveQuantizer &orig, int bbs = 32)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
void init_code_packer()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void range_search_dispatch_implem(idx_t n, const float *x, float radius, RangeSearchResult &rres, const CoarseQuantized &cq_in, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_10(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_12(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, int impl, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual CodePacker *get_CodePacker() const override
-
void reconstruct_orig_invlists()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFLocalSearchQuantizerFastScan : public faiss::IndexIVFAdditiveQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFLocalSearchQuantizerFastScan(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
-
IndexIVFLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
void init_code_packer()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void range_search_dispatch_implem(idx_t n, const float *x, float radius, RangeSearchResult &rres, const CoarseQuantized &cq_in, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_10(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_12(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, int impl, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual CodePacker *get_CodePacker() const override
-
void reconstruct_orig_invlists()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFResidualQuantizerFastScan : public faiss::IndexIVFAdditiveQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFResidualQuantizerFastScan(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
-
IndexIVFResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
void init_code_packer()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void range_search_dispatch_implem(idx_t n, const float *x, float radius, RangeSearchResult &rres, const CoarseQuantized &cq_in, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_10(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_12(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, int impl, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual CodePacker *get_CodePacker() const override
-
void reconstruct_orig_invlists()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFProductLocalSearchQuantizerFastScan : public faiss::IndexIVFAdditiveQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFProductLocalSearchQuantizerFastScan(Index *quantizer, size_t d, size_t nlist, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
-
IndexIVFProductLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
void init_code_packer()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void range_search_dispatch_implem(idx_t n, const float *x, float radius, RangeSearchResult &rres, const CoarseQuantized &cq_in, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_10(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_12(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, int impl, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual CodePacker *get_CodePacker() const override
-
void reconstruct_orig_invlists()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFProductResidualQuantizerFastScan : public faiss::IndexIVFAdditiveQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFProductResidualQuantizerFastScan(Index *quantizer, size_t d, size_t nlist, size_t nsplits, size_t Msub, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
-
IndexIVFProductResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
void init_code_packer()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void range_search_dispatch_implem(idx_t n, const float *x, float radius, RangeSearchResult &rres, const CoarseQuantized &cq_in, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_10(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_12(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, int impl, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual CodePacker *get_CodePacker() const override
-
void reconstruct_orig_invlists()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFFastScan : public faiss::IndexIVF
- #include <IndexIVFFastScan.h>
Fast scan version of IVFPQ and IVFAQ. Works for 4-bit PQ/AQ for now.
The codes in the inverted lists are not stored sequentially but grouped in blocks of size bbs. This makes it possible to very quickly compute distances with SIMD instructions.
Implementations (implem): 0: auto-select implementation (default) 1: orig’s search, re-implemented 2: orig’s search, re-ordered by invlist 10: optimizer int16 search, collect results in heap, no qbs 11: idem, collect results in reservoir 12: optimizer int16 search, collect results in heap, uses qbs 13: idem, collect results in reservoir 14: internally multithreaded implem over nq * nprobe 15: same with reservoir
For range search, only 10 and 12 are supported. add 100 to the implem to force single-thread scanning (the coarse quantizer may still use multiple threads).
Subclassed by faiss::IndexIVFAdditiveQuantizerFastScan, faiss::IndexIVFPQFastScan
Public Functions
-
IndexIVFFastScan(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
-
IndexIVFFastScan()
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
void init_code_packer()
-
~IndexIVFFastScan() override
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual bool lookup_table_is_3d() const = 0
-
virtual void compute_LUT(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const = 0
-
void compute_LUT_uint8(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void range_search_dispatch_implem(idx_t n, const float *x, float radius, RangeSearchResult &rres, const CoarseQuantized &cq_in, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_10(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_12(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, int impl, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual CodePacker *get_CodePacker() const override
-
void reconstruct_orig_invlists()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const = 0
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign)
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
struct CoarseQuantized
-
IndexIVFFastScan(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
-
struct IVFFastScanStats
Public Functions
-
inline double Mcy_at(int i)
-
inline double Mcy_reservoir_at(int i)
-
inline IVFFastScanStats()
-
inline void reset()
Public Members
-
uint64_t times[10]
-
uint64_t t_compute_distance_tables
-
uint64_t t_round
-
uint64_t t_copy_pack
-
uint64_t t_scan
-
uint64_t t_to_flat
-
uint64_t reservoir_times[4]
-
double t_aq_encode
-
double t_aq_norm_encode
-
inline double Mcy_at(int i)
-
struct IndexIVFFlat : public faiss::IndexIVF
- #include <IndexIVFFlat.h>
Inverted file with stored vectors. Here the inverted file pre-selects the vectors to be searched, but they are not otherwise encoded, the code array just contains the raw float entries.
Subclassed by faiss::IndexIVFFlatDedup
Public Functions
-
IndexIVFFlat(Index *quantizer, size_t d, size_t nlist_, MetricType = METRIC_L2)
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
IndexIVFFlat()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign)
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
IndexIVFFlat(Index *quantizer, size_t d, size_t nlist_, MetricType = METRIC_L2)
-
struct IndexIVFFlatDedup : public faiss::IndexIVFFlat
-
Public Functions
-
IndexIVFFlatDedup(Index *quantizer, size_t d, size_t nlist_, MetricType = METRIC_L2)
-
virtual void train(idx_t n, const float *x) override
also dedups the training set
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
implemented for all IndexIVF* classes
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
not implemented
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v) override
not implemented
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
not implemented
-
inline IndexIVFFlatDedup()
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign)
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
std::unordered_multimap<idx_t, idx_t> instances
Maps ids stored in the index to the ids of vectors that are the same. When a vector is unique, it does not appear in the instances map
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
IndexIVFFlatDedup(Index *quantizer, size_t d, size_t nlist_, MetricType = METRIC_L2)
-
struct IndexIVFIndependentQuantizer : public faiss::Index
- #include <IndexIVFIndependentQuantizer.h>
An IVF index with a quantizer that has a different input dimension from the payload size. The vectors to encode are obtained from the input vectors by a VectorTransform.
Public Functions
-
IndexIVFIndependentQuantizer(Index *quantizer, IndexIVF *index_ivf, VectorTransform *vt = nullptr)
-
inline IndexIVFIndependentQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reset() override
removes all elements from the database.
-
~IndexIVFIndependentQuantizer() override
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
Index *quantizer = nullptr
quantizer is fed directly with the input vectors
-
VectorTransform *vt = nullptr
transform before the IVF vectors are applied
-
IndexIVF *index_ivf = nullptr
the IVF index, controls nlist and nprobe
-
bool own_fields = false
whether *this owns the 3 fields
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexIVFIndependentQuantizer(Index *quantizer, IndexIVF *index_ivf, VectorTransform *vt = nullptr)
-
struct IVFPQSearchParameters : public faiss::SearchParametersIVF
Public Functions
-
inline IVFPQSearchParameters()
-
inline ~IVFPQSearchParameters()
Public Members
-
size_t scan_table_threshold
use table computation or on-the-fly?
-
int polysemous_ht
Hamming thresh for polysemous filtering.
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
SearchParameters *quantizer_params = nullptr
-
void *inverted_list_context = nullptr
context object to pass to InvertedLists
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
inline IVFPQSearchParameters()
-
struct IndexIVFPQ : public faiss::IndexIVF
- #include <IndexIVFPQ.h>
Inverted file with Product Quantizer encoding. Each residual vector is encoded as a product quantizer code.
Subclassed by faiss::IndexIVFPQR
Public Functions
-
IndexIVFPQ(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits_per_idx, MetricType metric = METRIC_L2)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_core_o(idx_t n, const float *x, const idx_t *xids, float *residuals_2, const idx_t *precomputed_idx = nullptr, void *inverted_list_context = nullptr)
same as add_core, also:
output 2nd level residuals if residuals_2 != NULL
accepts precomputed_idx = nullptr
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
trains the product quantizer
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
size_t find_duplicates(idx_t *ids, size_t *lims) const
Find exact duplicates in the dataset.
the duplicates are returned in pre-allocated arrays (see the max sizes).
- Parameters:
lims – limits between groups of duplicates (max size ntotal / 2 + 1)
ids – ids[lims[i]] : ids[lims[i+1]-1] is a group of duplicates (max size ntotal)
- Returns:
n number of groups found
-
void encode(idx_t key, const float *x, uint8_t *code) const
-
void encode_multiple(size_t n, idx_t *keys, const float *x, uint8_t *codes, bool compute_keys = false) const
Encode multiple vectors
- Parameters:
n – nb vectors to encode
keys – posting list ids for those vectors (size n)
x – vectors (size n * d)
codes – output codes (size n * code_size)
compute_keys – if false, assume keys are precomputed, otherwise compute them
-
void decode_multiple(size_t n, const idx_t *keys, const uint8_t *xcodes, float *x) const
inverse of encode_multiple
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
void precompute_table()
build precomputed table
-
IndexIVFPQ()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
ProductQuantizer pq
produces the codes
-
bool do_polysemous_training
reorder PQ centroids after training?
-
PolysemousTraining *polysemous_training
if NULL, use default
-
size_t scan_table_threshold
use table computation or on-the-fly?
-
int polysemous_ht
Hamming thresh for polysemous filtering.
-
int use_precomputed_table
Precompute table that speed up query preprocessing at some memory cost (used only for by_residual with L2 metric)
-
AlignedTable<float> precomputed_table
if use_precompute_table size nlist * pq.M * pq.ksub
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
IndexIVFPQ(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits_per_idx, MetricType metric = METRIC_L2)
-
struct IndexIVFPQStats
- #include <IndexIVFPQ.h>
statistics are robust to internal threading, but not if IndexIVFPQ::search_preassigned is called by multiple threads
Public Functions
-
inline IndexIVFPQStats()
-
void reset()
Public Members
-
size_t nrefine
nb of refines (IVFPQR)
-
size_t n_hamming_pass
nb of passed Hamming distance tests (for polysemous)
-
size_t search_cycles
-
size_t refine_cycles
only for IVFPQR
-
inline IndexIVFPQStats()
-
struct IndexIVFPQFastScan : public faiss::IndexIVFFastScan
- #include <IndexIVFPQFastScan.h>
Fast scan version of IVFPQ. Works for 4-bit PQ for now.
The codes in the inverted lists are not stored sequentially but grouped in blocks of size bbs. This makes it possible to very quickly compute distances with SIMD instructions.
Implementations (implem): 0: auto-select implementation (default) 1: orig’s search, re-implemented 2: orig’s search, re-ordered by invlist 10: optimizer int16 search, collect results in heap, no qbs 11: idem, collect results in reservoir 12: optimizer int16 search, collect results in heap, uses qbs 13: idem, collect results in reservoir
Public Functions
-
IndexIVFPQFastScan(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, int bbs = 32)
-
IndexIVFPQFastScan()
-
explicit IndexIVFPQFastScan(const IndexIVFPQ &orig, int bbs = 32)
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
void precompute_table()
build precomputed table, possibly updating use_precomputed_table
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFPQ encoder. The codes are not reorganized by blocks a that point
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
void init_code_packer()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const CoarseQuantized &cq, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void range_search_dispatch_implem(idx_t n, const float *x, float radius, RangeSearchResult &rres, const CoarseQuantized &cq_in, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
template<class C>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_10(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_12(idx_t n, const float *x, SIMDResultHandlerToFloat &handler, const CoarseQuantized &cq, size_t *ndis_out, size_t *nlist_out, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const CoarseQuantized &cq, int impl, const NormTableScaler *scaler, const IVFSearchParameters *params = nullptr) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual CodePacker *get_CodePacker() const override
-
void reconstruct_orig_invlists()
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false, const IDSelector *sel = nullptr) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
ProductQuantizer pq
produces the codes
-
int use_precomputed_table = 0
precomputed tables management
-
AlignedTable<float> precomputed_table
if use_precompute_table size (nlist, pq.M, pq.ksub)
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
IndexIVFPQFastScan(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, int bbs = 32)
-
struct IndexIVFPQR : public faiss::IndexIVFPQ
- #include <IndexIVFPQR.h>
Index with an additional level of PQ refinement
Public Functions
-
IndexIVFPQR(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits_per_idx, size_t M_refine, size_t nbits_per_idx_refine)
-
virtual void reset() override
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
trains the two product quantizers
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr) override
same as add_with_ids, but optionally use the precomputed list ids
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
IndexIVFPQR()
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void add_core_o(idx_t n, const float *x, const idx_t *xids, float *residuals_2, const idx_t *precomputed_idx = nullptr, void *inverted_list_context = nullptr)
same as add_core, also:
output 2nd level residuals if residuals_2 != NULL
accepts precomputed_idx = nullptr
-
size_t find_duplicates(idx_t *ids, size_t *lims) const
Find exact duplicates in the dataset.
the duplicates are returned in pre-allocated arrays (see the max sizes).
- Parameters:
lims – limits between groups of duplicates (max size ntotal / 2 + 1)
ids – ids[lims[i]] : ids[lims[i+1]-1] is a group of duplicates (max size ntotal)
- Returns:
n number of groups found
-
void encode_multiple(size_t n, idx_t *keys, const float *x, uint8_t *codes, bool compute_keys = false) const
Encode multiple vectors
- Parameters:
n – nb vectors to encode
keys – posting list ids for those vectors (size n)
x – vectors (size n * d)
codes – output codes (size n * code_size)
compute_keys – if false, assume keys are precomputed, otherwise compute them
-
void decode_multiple(size_t n, const idx_t *keys, const uint8_t *xcodes, float *x) const
inverse of encode_multiple
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
void precompute_table()
build precomputed table
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
ProductQuantizer refine_pq
3rd level quantizer
-
std::vector<uint8_t> refine_codes
corresponding codes
-
float k_factor
factor between k requested in search and the k requested from the IVFPQ
-
ProductQuantizer pq
produces the codes
-
bool do_polysemous_training
reorder PQ centroids after training?
-
PolysemousTraining *polysemous_training
if NULL, use default
-
size_t scan_table_threshold
use table computation or on-the-fly?
-
int polysemous_ht
Hamming thresh for polysemous filtering.
-
int use_precomputed_table
Precompute table that speed up query preprocessing at some memory cost (used only for by_residual with L2 metric)
-
AlignedTable<float> precomputed_table
if use_precompute_table size nlist * pq.M * pq.ksub
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
IndexIVFPQR(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits_per_idx, size_t M_refine, size_t nbits_per_idx_refine)
-
struct IndexIVFSpectralHash : public faiss::IndexIVF
- #include <IndexIVFSpectralHash.h>
Inverted list that stores binary codes of size nbit. Before the binary conversion, the dimension of the vectors is transformed from dim d into dim nbit by vt (a random rotation by default).
Each coordinate is subtracted from a value determined by threshold_type, and split into intervals of size period. Half of the interval is a 0 bit, the other half a 1.
Public Types
-
enum ThresholdType
Values:
-
enumerator Thresh_global
global threshold at 0
-
enumerator Thresh_centroid
compare to centroid
-
enumerator Thresh_centroid_half
central interval around centroid
-
enumerator Thresh_median
median of training set
-
enumerator Thresh_global
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexIVFSpectralHash(Index *quantizer, size_t d, size_t nlist, int nbit, float period)
-
IndexIVFSpectralHash()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
void replace_vt(VectorTransform *vt, bool own = false)
replace the vector transform for an empty (and possibly untrained) index
-
void replace_vt(IndexPreTransform *index, bool own = false)
convenience function to get the VT from an index constucted by an index_factory (should end in “LSH”)
-
~IndexIVFSpectralHash() override
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual idx_t train_encoder_num_vectors() const
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
VectorTransform *vt = nullptr
transformation from d to nbit dim
-
bool own_fields = true
own the vt
-
int nbit = 0
nb of bits of the binary signature
-
float period = 0
interval size for 0s and 1s
-
ThresholdType threshold_type = Thresh_global
-
std::vector<float> trained
Trained threshold. size nlist * nbit or 0 if Thresh_global
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
ClusteringParameters cp
to override default clustering params
-
enum ThresholdType
-
struct IndexLattice : public faiss::Index
- #include <IndexLattice.h>
Index that encodes a vector with a series of Zn lattice quantizers
Public Functions
-
IndexLattice(idx_t d, int nsq, int scale_nbit, int r2)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void add(idx_t n, const float *x) override
not implemented
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
Public Members
-
int nsq
number of sub-vectors
-
size_t dsq
dimension of sub-vectors
-
ZnSphereCodecAlt zn_sphere_codec
the lattice quantizer
-
int scale_nbit
nb bits used to encode the scale, per subvector
-
int lattice_nbit
-
size_t code_size
total, in bytes
-
std::vector<float> trained
mins and maxes of the vector norms, per subquantizer
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexLattice(idx_t d, int nsq, int scale_nbit, int r2)
-
struct IndexLSH : public faiss::IndexFlatCodes
- #include <IndexLSH.h>
The sign of each vector component is put in a binary signature
Public Functions
-
IndexLSH(idx_t d, int nbits, bool rotate_data = true, bool train_thresholds = false)
-
const float *apply_preprocess(idx_t n, const float *x) const
Preprocesses and resizes the input to the size required to binarize the data
- Parameters:
x – input vectors, size n * d
- Returns:
output vectors, size n * bits. May be the same pointer as x, otherwise it should be deleted by the caller
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
void transfer_thresholds(LinearTransform *vt)
transfer the thresholds to a pre-processing stage (and unset train_thresholds)
-
inline ~IndexLSH() override
-
IndexLSH()
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const
a FlatCodesDistanceComputer offers a distance_to_code method
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
int nbits
nb of bits per vector
-
bool rotate_data
whether to apply a random rotation to input
-
bool train_thresholds
whether we train thresholds or use 0
-
RandomRotationMatrix rrot
optional random rotation
-
std::vector<float> thresholds
thresholds to compare with
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexLSH(idx_t d, int nbits, bool rotate_data = true, bool train_thresholds = false)
-
struct IndexNNDescent : public faiss::Index
- #include <IndexNNDescent.h>
The NNDescent index is a normal random-access index with an NNDescent link structure built on top
Subclassed by faiss::IndexNNDescentFlat
Public Types
-
using storage_idx_t = NNDescent::storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
explicit IndexNNDescent(int d = 0, int K = 32, MetricType metric = METRIC_L2)
-
explicit IndexNNDescent(Index *storage, int K = 32)
-
~IndexNNDescent() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
using storage_idx_t = NNDescent::storage_idx_t
-
struct IndexNNDescentFlat : public faiss::IndexNNDescent
- #include <IndexNNDescent.h>
Flat index topped with with a NNDescent structure to access elements more efficiently.
Public Types
-
using storage_idx_t = NNDescent::storage_idx_t
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexNNDescentFlat()
-
IndexNNDescentFlat(int d, int K, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
using storage_idx_t = NNDescent::storage_idx_t
-
struct IndexNSG : public faiss::Index
- #include <IndexNSG.h>
The NSG index is a normal random-access index with a NSG link structure built on top
Subclassed by faiss::IndexNSGFlat, faiss::IndexNSGPQ, faiss::IndexNSGSQ
Public Functions
-
explicit IndexNSG(int d = 0, int R = 32, MetricType metric = METRIC_L2)
-
explicit IndexNSG(Index *storage, int R = 32)
-
~IndexNSG() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields = false
the sequential storage
-
Index *storage = nullptr
-
bool is_built = false
the index is built or not
-
int GK = 64
K of KNN graph for building.
-
char build_type = 0
indicate how to build a knn graph
-
int nndescent_S = 10
parameters for nndescent
-
int nndescent_R = 100
-
int nndescent_L
-
int nndescent_iter = 10
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
explicit IndexNSG(int d = 0, int R = 32, MetricType metric = METRIC_L2)
-
struct IndexNSGFlat : public faiss::IndexNSG
- #include <IndexNSG.h>
Flat index topped with with a NSG structure to access elements more efficiently.
Public Functions
-
IndexNSGFlat()
-
IndexNSGFlat(int d, int R, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields = false
the sequential storage
-
Index *storage = nullptr
-
bool is_built = false
the index is built or not
-
int GK = 64
K of KNN graph for building.
-
char build_type = 0
indicate how to build a knn graph
-
int nndescent_S = 10
parameters for nndescent
-
int nndescent_R = 100
-
int nndescent_L
-
int nndescent_iter = 10
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexNSGFlat()
-
struct IndexNSGPQ : public faiss::IndexNSG
- #include <IndexNSG.h>
PQ index topped with with a NSG structure to access elements more efficiently.
Public Functions
-
IndexNSGPQ()
-
IndexNSGPQ(int d, int pq_m, int M, int pq_nbits = 8)
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields = false
the sequential storage
-
Index *storage = nullptr
-
bool is_built = false
the index is built or not
-
int GK = 64
K of KNN graph for building.
-
char build_type = 0
indicate how to build a knn graph
-
int nndescent_S = 10
parameters for nndescent
-
int nndescent_R = 100
-
int nndescent_L
-
int nndescent_iter = 10
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexNSGPQ()
-
struct IndexNSGSQ : public faiss::IndexNSG
- #include <IndexNSG.h>
SQ index topped with with a NSG structure to access elements more efficiently.
Public Functions
-
IndexNSGSQ()
-
IndexNSGSQ(int d, ScalarQuantizer::QuantizerType qtype, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields = false
the sequential storage
-
Index *storage = nullptr
-
bool is_built = false
the index is built or not
-
int GK = 64
K of KNN graph for building.
-
char build_type = 0
indicate how to build a knn graph
-
int nndescent_S = 10
parameters for nndescent
-
int nndescent_R = 100
-
int nndescent_L
-
int nndescent_iter = 10
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexNSGSQ()
-
struct IndexPQ : public faiss::IndexFlatCodes
- #include <IndexPQ.h>
Index based on a product quantizer. Stored vectors are approximated by PQ codes.
Public Types
-
enum Search_type_t
how to perform the search in search_core
Values:
-
enumerator ST_PQ
asymmetric product quantizer (default)
-
enumerator ST_HE
Hamming distance on codes.
-
enumerator ST_generalized_HE
nb of same codes
-
enumerator ST_SDC
symmetric product quantizer (SDC)
-
enumerator ST_polysemous
HE filter (using ht) + PQ combination.
-
enumerator ST_polysemous_generalize
Filter on generalized Hamming.
-
enumerator ST_PQ
-
using component_t = float
-
using distance_t = float
Public Functions
-
IndexPQ(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexPQ()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
void search_core_polysemous(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int polysemous_ht, bool generalized_hamming) const
-
void hamming_distance_histogram(idx_t n, const float *x, idx_t nb, const float *xb, int64_t *dist_histogram)
prepare query for a polysemous search, but instead of computing the result, just get the histogram of Hamming distances. May be computed on a provided dataset if xb != NULL
- Parameters:
dist_histogram – (M * nbits + 1)
-
void hamming_distance_table(idx_t n, const float *x, int32_t *dis) const
compute pairwise distances between queries and database
- Parameters:
n – nb of query vectors
x – query vector, size n * d
dis – output distances, size n * ntotal
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ProductQuantizer pq
The product quantizer used to encode the vectors.
-
bool do_polysemous_training
false = standard PQ
-
PolysemousTraining polysemous_training
parameters used for the polysemous training
-
Search_type_t search_type
-
bool encode_signs
-
int polysemous_ht
Hamming threshold used for polysemy.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
enum Search_type_t
-
struct SearchParametersPQ : public faiss::SearchParameters
- #include <IndexPQ.h>
override search parameters from the class
Public Members
-
IndexPQ::Search_type_t search_type
-
int polysemous_ht
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
IndexPQ::Search_type_t search_type
-
struct IndexPQStats
- #include <IndexPQ.h>
statistics are robust to internal threading, but not if IndexPQ::search is called by multiple threads
Public Functions
-
inline IndexPQStats()
-
void reset()
Public Members
-
size_t nq
-
size_t ncode
-
size_t n_hamming_pass
-
inline IndexPQStats()
-
struct MultiIndexQuantizer : public faiss::Index
- #include <IndexPQ.h>
Quantizer where centroids are virtual: they are the Cartesian product of sub-centroids.
Subclassed by faiss::MultiIndexQuantizer2
Public Functions
-
MultiIndexQuantizer(int d, size_t M, size_t nbits)
number of bit per subvector index
- Parameters:
d – dimension of the input vectors
M – number of subquantizers
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void add(idx_t n, const float *x) override
add and reset will crash at runtime
-
virtual void reset() override
removes all elements from the database.
-
inline MultiIndexQuantizer()
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
MultiIndexQuantizer(int d, size_t M, size_t nbits)
-
struct MultiIndexQuantizer2 : public faiss::MultiIndexQuantizer
- #include <IndexPQ.h>
MultiIndexQuantizer where the PQ assignmnet is performed by sub-indexes
Public Functions
-
MultiIndexQuantizer2(int d, size_t M, size_t nbits, Index **indexes)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void add(idx_t n, const float *x) override
add and reset will crash at runtime
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
bool own_fields
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
MultiIndexQuantizer2(int d, size_t M, size_t nbits, Index **indexes)
-
struct IndexPQFastScan : public faiss::IndexFastScan
- #include <IndexPQFastScan.h>
Fast scan version of IndexPQ. Works for 4-bit PQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Public Functions
-
IndexPQFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, int bbs = 32)
-
IndexPQFastScan() = default
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexPQFastScan object are non-contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
template<bool is_max>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class Cfloat>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
template<class C>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const NormTableScaler *scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
CodePacker *get_CodePacker() const
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexPQFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, int bbs = 32)
-
struct SearchParametersPreTransform : public faiss::SearchParameters
Public Members
-
SearchParameters *index_params = nullptr
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
SearchParameters *index_params = nullptr
-
struct IndexPreTransform : public faiss::Index
- #include <IndexPreTransform.h>
Index that applies a LinearTransform transform on vectors before handing them over to a sub-index
Public Functions
-
explicit IndexPreTransform(Index *index)
! whether pointers are deleted in destructor
-
IndexPreTransform()
-
IndexPreTransform(VectorTransform *ltrans, Index *index)
ltrans is the last transform before the index
-
void prepend_transform(VectorTransform *ltrans)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void reset() override
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
const float *apply_chain(idx_t n, const float *x) const
apply the transforms in the chain. The returned float * may be equal to x, otherwise it should be deallocated.
-
void reverse_chain(idx_t n, const float *xt, float *x) const
Reverse the transforms in the chain. May not be implemented for all transforms in the chain or may return approximate results.
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
~IndexPreTransform() override
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
std::vector<VectorTransform*> chain
-
Index *index
! chain of tranforms
-
bool own_fields
! the sub-index
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
explicit IndexPreTransform(Index *index)
-
struct IndexRefineSearchParameters : public faiss::SearchParameters
Public Functions
-
virtual ~IndexRefineSearchParameters() = default
Public Members
-
float k_factor = 1
-
SearchParameters *base_index_params = nullptr
-
IDSelector *sel = nullptr
if non-null, only these IDs will be considered during search.
-
virtual ~IndexRefineSearchParameters() = default
-
struct IndexRefine : public faiss::Index
- #include <IndexRefine.h>
Index that queries in a base_index (a fast one) and refines the results with an exact search, hopefully improving the results.
Subclassed by faiss::IndexRefineFlat
Public Functions
-
IndexRefine()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
The sa_decode decodes from the index_refine, which is assumed to be more accurate
-
~IndexRefine() override
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
Public Members
-
Index *base_index
faster index to pre-select the vectors that should be filtered
-
Index *refine_index
refinement index
-
bool own_fields
should the base index be deallocated?
-
bool own_refine_index
same with the refinement index
-
float k_factor = 1
factor between k requested in search and the k requested from the base_index (should be >= 1)
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexRefine()
-
struct IndexRefineFlat : public faiss::IndexRefine
- #include <IndexRefine.h>
Version where the refinement index is an IndexFlat. It has one additional constructor that takes a table of elements to add to the flat refinement index
Public Functions
-
explicit IndexRefineFlat(Index *base_index)
-
IndexRefineFlat(Index *base_index, const float *xb)
-
IndexRefineFlat()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
The sa_decode decodes from the index_refine, which is assumed to be more accurate
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
Public Members
-
Index *base_index
faster index to pre-select the vectors that should be filtered
-
Index *refine_index
refinement index
-
bool own_fields
should the base index be deallocated?
-
bool own_refine_index
same with the refinement index
-
float k_factor = 1
factor between k requested in search and the k requested from the base_index (should be >= 1)
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
explicit IndexRefineFlat(Index *base_index)
-
template<typename IndexT>
class IndexReplicasTemplate : public faiss::ThreadedIndex<IndexT> - #include <IndexReplicas.h>
Takes individual faiss::Index instances, and splits queries for sending to each Index instance, and joins the results together when done. Each index is managed by a separate CPU thread.
Public Types
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexReplicasTemplate(bool threaded = true)
The dimension that all sub-indices must share will be the dimension of the first sub-index added
- Parameters:
threaded – do we use one thread per sub-index or do queries sequentially?
-
explicit IndexReplicasTemplate(idx_t d, bool threaded = true)
- Parameters:
d – the dimension that all sub-indices must share
threaded – do we use one thread per sub index or do queries sequentially?
-
explicit IndexReplicasTemplate(int d, bool threaded = true)
int version due to the implicit bool conversion ambiguity of int as dimension
-
inline void add_replica(IndexT *index)
Alias for addIndex()
-
inline void remove_replica(IndexT *index)
Alias for removeIndex()
-
void train(idx_t n, const component_t *x) override
faiss::Index API All indices receive the same call
-
void add(idx_t n, const component_t *x) override
faiss::Index API All indices receive the same call
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
faiss::Index API Query is partitioned into a slice for each sub-index split by ceil(n / #indices) for our sub-indices
-
void reconstruct(idx_t, component_t *v) const override
reconstructs from the first index
-
void syncWithSubIndexes()
Synchronize the top-level index (IndexShards) with data in the sub-indices
-
virtual void addIndex(IndexT *index)
override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.
-
void removeIndex(IndexT *index)
Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.
-
void runOnIndex(std::function<void(int, IndexT*)> f)
Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)
-
void reset() override
faiss::Index API All indices receive the same call
-
inline int count() const
Returns the number of sub-indices.
-
inline IndexT *at(size_t i)
Returns the i-th sub-index.
-
inline const IndexT *at(size_t i) const
Returns the i-th sub-index (const version)
Public Members
-
bool own_indices = false
Whether or not we are responsible for deleting our contained indices.
Protected Functions
-
virtual void onAfterAddIndex(IndexT *index) override
Called just after an index is added.
-
virtual void onAfterRemoveIndex(IndexT *index) override
Called just after an index is removed.
-
using component_t = typename IndexT::component_t
-
struct IndexRowwiseMinMaxBase : public faiss::Index
- #include <IndexRowwiseMinMax.h>
Provides base functions for rowwise normalizing indices.
Index wrapper that performs rowwise normalization to [0,1], preserving the coefficients. This is a vector codec index only.
Basically, this index performs a rowwise scaling to [0,1] of every row in an input dataset before calling subindex::train() and subindex::sa_encode(). sa_encode() call stores the scaling coefficients (scaler and minv) in the very beginning of every output code. The format: [scaler][minv][subindex::sa_encode() output] The de-scaling in sa_decode() is done using: output_rescaled = scaler * output + minv
An additional ::train_inplace() function is provided in order to do an inplace scaling before calling subindex::train() and, thus, avoiding the cloning of the input dataset, but modifying the input dataset because of the scaling and the scaling back. It is up to user to call this function instead of ::train()
Derived classes provide different data types for scaling coefficients. Currently, versions with fp16 and fp32 scaling coefficients are available.
fp16 version adds 4 extra bytes per encoded vector
fp32 version adds 8 extra bytes per encoded vector
Subclassed by faiss::IndexRowwiseMinMax, faiss::IndexRowwiseMinMaxFP16
Public Functions
-
explicit IndexRowwiseMinMaxBase(Index *index)
-
IndexRowwiseMinMaxBase()
-
~IndexRowwiseMinMaxBase() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reset() override
removes all elements from the database.
-
virtual void train_inplace(idx_t n, float *x) = 0
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
struct IndexRowwiseMinMaxFP16 : public faiss::IndexRowwiseMinMaxBase
- #include <IndexRowwiseMinMax.h>
Stores scaling coefficients as fp16 values.
Public Functions
-
explicit IndexRowwiseMinMaxFP16(Index *index)
-
IndexRowwiseMinMaxFP16()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void train_inplace(idx_t n, float *x) override
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
explicit IndexRowwiseMinMaxFP16(Index *index)
-
struct IndexRowwiseMinMax : public faiss::IndexRowwiseMinMaxBase
- #include <IndexRowwiseMinMax.h>
Stores scaling coefficients as fp32 values.
Public Functions
-
explicit IndexRowwiseMinMax(Index *index)
-
IndexRowwiseMinMax()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void train_inplace(idx_t n, float *x) override
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
explicit IndexRowwiseMinMax(Index *index)
-
struct IndexScalarQuantizer : public faiss::IndexFlatCodes
- #include <IndexScalarQuantizer.h>
Flat index built on a scalar quantizer.
Public Functions
-
IndexScalarQuantizer(int d, ScalarQuantizer::QuantizerType qtype, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexScalarQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that because of the structure of the index, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
CodePacker *get_CodePacker() const
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ScalarQuantizer sq
Used to encode the vectors.
-
size_t code_size
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
IndexScalarQuantizer(int d, ScalarQuantizer::QuantizerType qtype, MetricType metric = METRIC_L2)
-
struct IndexIVFScalarQuantizer : public faiss::IndexIVF
- #include <IndexScalarQuantizer.h>
An IVF implementation where the components of the residuals are encoded with a scalar quantizer. All distance computations are asymmetric, so the encoded vectors are decoded and approximate distances are computed.
Public Functions
-
IndexIVFScalarQuantizer(Index *quantizer, size_t d, size_t nlist, ScalarQuantizer::QuantizerType qtype, MetricType metric = METRIC_L2, bool by_residual = true)
-
IndexIVFScalarQuantizer()
-
virtual void train_encoder(idx_t n, const float *x, const idx_t *assign) override
Train the encoder for the vectors.
If by_residual then it is called with residuals and corresponding assign array, otherwise x is the raw training vectors and assign=nullptr
-
virtual idx_t train_encoder_num_vectors() const override
can be redefined by subclasses to indicate how many training vectors they need
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx, void *inverted_list_context = nullptr) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs, const IDSelector *sel) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_encoder to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void search_and_return_codes(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, uint8_t *recons, bool include_listno = false, const SearchParameters *params = nullptr) const
Similar to search, but also returns the codes corresponding to the stored vectors for the search results.
- Parameters:
codes – codes (n, k, code_size)
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void check_compatible_for_merge(const Index &otherIndex) const override
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(Index &otherIndex, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual CodePacker *get_CodePacker() const
-
virtual void copy_subset_to(IndexIVF &other, InvertedLists::subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index see Invlists::copy_subset_to for the meaning of subset_type
-
inline size_t get_list_size(size_t list_no) const
-
bool check_ids_sorted() const
are the ids sorted?
-
void make_direct_map(bool new_maintain_direct_map = true)
initialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors sa_encode will call encode_vector with include_listno=true
- Parameters:
n – nb of vectors to encode
x – the vectors to encode
bytes – output array for the codes
- Returns:
nb of bytes written to codes
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
InvertedLists *invlists = nullptr
Access to the actual data.
-
bool own_invlists = false
-
size_t code_size = 0
code size per vector in bytes
-
int parallel_mode = 0
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
bool by_residual = true
do the codes in the invlists encode the vectors relative to the centroids?
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
IndexIVFScalarQuantizer(Index *quantizer, size_t d, size_t nlist, ScalarQuantizer::QuantizerType qtype, MetricType metric = METRIC_L2, bool by_residual = true)
-
template<typename IndexT>
struct IndexShardsTemplate : public faiss::ThreadedIndex<IndexT> - #include <IndexShards.h>
Index that concatenates the results from several sub-indexes
Subclassed by faiss::IndexShardsIVF
Public Types
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexShardsTemplate(bool threaded = false, bool successive_ids = true)
The dimension that all sub-indices must share will be the dimension of the first sub-index added
- Parameters:
threaded – do we use one thread per sub_index or do queries sequentially?
successive_ids – should we shift the returned ids by the size of each sub-index or return them as they are?
-
explicit IndexShardsTemplate(idx_t d, bool threaded = false, bool successive_ids = true)
- Parameters:
threaded – do we use one thread per sub_index or do queries sequentially?
successive_ids – should we shift the returned ids by the size of each sub-index or return them as they are?
-
explicit IndexShardsTemplate(int d, bool threaded = false, bool successive_ids = true)
int version due to the implicit bool conversion ambiguity of int as dimension
-
inline void add_shard(IndexT *index)
Alias for addIndex()
-
inline void remove_shard(IndexT *index)
Alias for removeIndex()
-
void add(idx_t n, const component_t *x) override
supported only for sub-indices that implement add_with_ids
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
Cases (successive_ids, xids):
true, non-NULL ERROR: it makes no sense to pass in ids and request them to be shifted
true, NULL OK: but should be called only once (calls add() on sub-indexes).
false, non-NULL OK: will call add_with_ids with passed in xids distributed evenly over shards
false, NULL OK: will call add_with_ids on each sub-index, starting at ntotal
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
-
void train(idx_t n, const component_t *x) override
-
virtual void syncWithSubIndexes()
Synchronize the top-level index (IndexShards) with data in the sub-indices
-
virtual void addIndex(IndexT *index)
override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.
-
void removeIndex(IndexT *index)
Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.
-
void runOnIndex(std::function<void(int, IndexT*)> f)
Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)
-
void reset() override
faiss::Index API All indices receive the same call
-
inline int count() const
Returns the number of sub-indices.
Public Members
-
bool successive_ids
-
bool own_indices = false
Whether or not we are responsible for deleting our contained indices.
Protected Functions
-
virtual void onAfterAddIndex(IndexT *index) override
Called just after an index is added.
-
virtual void onAfterRemoveIndex(IndexT *index) override
Called just after an index is removed.
-
using component_t = typename IndexT::component_t
-
struct IndexShardsIVF : public faiss::IndexShardsTemplate<Index>, public faiss::Level1Quantizer
- #include <IndexShardsIVF.h>
IndexShards with a common coarse quantizer. All the indexes added should be IndexIVFInterface indexes so that the search_precomputed can be called.
Public Types
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexShardsIVF(Index *quantizer, size_t nlist, bool threaded = false, bool successive_ids = true)
-
void addIndex(Index *index) override
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
-
void train(idx_t n, const component_t *x) override
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
-
inline void add_shard(IndexT *index)
Alias for addIndex()
-
inline void remove_shard(IndexT *index)
Alias for removeIndex()
-
void add(idx_t n, const component_t *x) override
supported only for sub-indices that implement add_with_ids
-
virtual void syncWithSubIndexes()
Synchronize the top-level index (IndexShards) with data in the sub-indices
-
virtual void addIndex(IndexT *index)
override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.
-
void removeIndex(IndexT *index)
Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.
-
void runOnIndex(std::function<void(int, IndexT*)> f)
Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)
-
void reset() override
faiss::Index API All indices receive the same call
-
inline int count() const
Returns the number of sub-indices.
-
inline IndexT *at(size_t i)
Returns the i-th sub-index.
-
inline const IndexT *at(size_t i) const
Returns the i-th sub-index (const version)
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
bool successive_ids
-
bool own_indices = false
Whether or not we are responsible for deleting our contained indices.
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
Protected Functions
-
virtual void onAfterAddIndex(IndexT *index) override
Called just after an index is added.
-
virtual void onAfterRemoveIndex(IndexT *index) override
Called just after an index is removed.
-
using component_t = typename IndexT::component_t
-
struct BlockInvertedLists : public faiss::InvertedLists
- #include <BlockInvertedLists.h>
Inverted Lists that are organized by blocks.
Different from the regular inverted lists, the codes are organized by blocks of size block_size bytes that reprsent a set of n_per_block. Therefore, code allocations are always rounded up to block_size bytes. The codes are also aligned on 32-byte boundaries for use with SIMD.
To avoid misinterpretations, the code_size is set to (size_t)(-1), even if arguably the amount of memory consumed by code is block_size / n_per_block.
The writing functions add_entries and update_entries operate on block-aligned data.
Public Types
Public Functions
-
BlockInvertedLists(size_t nlist, size_t vec_per_block, size_t block_size)
-
BlockInvertedLists(size_t nlist, const CodePacker *packer)
-
BlockInvertedLists()
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
size_t remove_ids(const IDSelector &sel)
remove ids from the InvertedLists
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
not implemented
-
virtual void resize(size_t list_no, size_t new_size) override
-
~BlockInvertedLists() override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
size_t n_per_block = 0
-
size_t block_size = 0
-
const CodePacker *packer = nullptr
-
std::vector<AlignedTable<uint8_t>> codes
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
BlockInvertedLists(size_t nlist, size_t vec_per_block, size_t block_size)
-
struct BlockInvertedListsIOHook : public faiss::InvertedListsIOHook
Public Functions
-
BlockInvertedListsIOHook()
-
virtual void write(const InvertedLists *ils, IOWriter *f) const override
write the index to the IOWriter (including the fourcc)
-
virtual InvertedLists *read(IOReader *f, int io_flags) const override
called when the fourcc matches this class’s fourcc
-
virtual InvertedLists *read_ArrayInvertedLists(IOReader *f, int io_flags, size_t nlist, size_t code_size, const std::vector<size_t> &sizes) const
read from a ArrayInvertedLists into this invertedlist type. For this to work, the callback has to be enabled and the io_flag has to be set to IO_FLAG_SKIP_IVF_DATA | (16 upper bits of the fourcc)
(default implementation fails)
Public Members
Public Static Functions
-
static void add_callback(InvertedListsIOHook*)
-
static void print_callbacks()
-
static InvertedListsIOHook *lookup(int h)
-
static InvertedListsIOHook *lookup_classname(const std::string &classname)
-
BlockInvertedListsIOHook()
-
struct DirectMap
- #include <DirectMap.h>
Direct map: a way to map back from ids to inverted lists
Public Types
-
enum Type
Values:
-
enumerator NoMap
-
enumerator Array
-
enumerator Hashtable
-
enumerator NoMap
Public Functions
-
DirectMap()
-
void set_type(Type new_type, const InvertedLists *invlists, size_t ntotal)
set type and initialize
-
inline bool no() const
for quick checks
-
void check_can_add(const idx_t *ids)
throw if Array and ids is not NULL
update the direct_map
-
void clear()
remove all entries
-
size_t remove_ids(const IDSelector &sel, InvertedLists *invlists)
remove ids from the InvertedLists, possibly using the direct map
operations on inverted lists that require translation with a DirectMap
-
void update_codes(InvertedLists *invlists, int n, const idx_t *ids, const idx_t *list_nos, const uint8_t *codes)
update entries, using the direct map
-
enum Type
-
struct DirectMapAdd
- #include <DirectMap.h>
Thread-safe way of updating the direct_map.
Public Functions
-
void add(size_t i, idx_t list_no, size_t offset)
add vector i (with id xids[i]) at list_no and offset
-
~DirectMapAdd()
-
void add(size_t i, idx_t list_no, size_t offset)
-
struct InvertedListsIterator
-
struct InvertedLists
- #include <InvertedLists.h>
Table of inverted lists multithreading rules:
concurrent read accesses are allowed
concurrent update accesses are allowed
for resize and add_entries, only concurrent access to different lists are allowed
Subclassed by faiss::ArrayInvertedLists, faiss::BlockInvertedLists, faiss::OnDiskInvertedLists, faiss::ReadOnlyInvertedLists
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
InvertedLists(size_t nlist, size_t code_size)
-
virtual ~InvertedLists()
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual size_t list_size(size_t list_no) const = 0
get the size of a list
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual const uint8_t *get_codes(size_t list_no) const = 0
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const = 0
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) = 0
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) = 0
-
virtual void resize(size_t list_no, size_t new_size) = 0
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
struct ScopedCodes
Public Functions
-
inline ScopedCodes(const InvertedLists *il, size_t list_no)
-
inline ScopedCodes(const InvertedLists *il, size_t list_no, size_t offset)
-
inline const uint8_t *get()
-
inline ~ScopedCodes()
-
inline ScopedCodes(const InvertedLists *il, size_t list_no)
-
struct ScopedIds
Public Functions
-
inline ScopedIds(const InvertedLists *il, size_t list_no)
-
inline const idx_t *get()
-
inline idx_t operator[](size_t i) const
-
inline ~ScopedIds()
-
inline ScopedIds(const InvertedLists *il, size_t list_no)
-
struct ArrayInvertedLists : public faiss::InvertedLists
- #include <InvertedLists.h>
simple (default) implementation as an array of inverted lists
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
ArrayInvertedLists(size_t nlist, size_t code_size)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
void permute_invlists(const idx_t *map)
permute the inverted lists, map maps new_id to old_id
-
~ArrayInvertedLists() override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
enum subset_type_t
-
struct ReadOnlyInvertedLists : public faiss::InvertedLists
- #include <InvertedLists.h>
invlists that fail for all write functions
Subclassed by faiss::HStackInvertedLists, faiss::MaskedInvertedLists, faiss::SliceInvertedLists, faiss::StopWordsInvertedLists, faiss::VStackInvertedLists
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
inline ReadOnlyInvertedLists(size_t nlist, size_t code_size)
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual size_t list_size(size_t list_no) const = 0
get the size of a list
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual const uint8_t *get_codes(size_t list_no) const = 0
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const = 0
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
enum subset_type_t
-
struct HStackInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
Horizontal stack of inverted lists.
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
HStackInvertedLists(int nil, const InvertedLists **ils)
build InvertedLists by concatenating nil of them
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
std::vector<const InvertedLists*> ils
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
enum subset_type_t
-
struct SliceInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
vertical slice of indexes in another InvertedLists
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
SliceInvertedLists(const InvertedLists *il, idx_t i0, idx_t i1)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
const InvertedLists *il
-
idx_t i0
-
idx_t i1
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
enum subset_type_t
-
struct VStackInvertedLists : public faiss::ReadOnlyInvertedLists
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
VStackInvertedLists(int nil, const InvertedLists **ils)
build InvertedLists by concatenating nil of them
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
std::vector<const InvertedLists*> ils
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
enum subset_type_t
-
struct MaskedInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
use the first inverted lists if they are non-empty otherwise use the second
This is useful if il1 has a few inverted lists that are too long, and that il0 has replacement lists for those, with empty lists for the others.
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
MaskedInvertedLists(const InvertedLists *il0, const InvertedLists *il1)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
const InvertedLists *il0
-
const InvertedLists *il1
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
enum subset_type_t
-
struct StopWordsInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
if the inverted list in il is smaller than maxsize then return it, otherwise return an empty invlist
Public Types
-
enum subset_type_t
Values:
-
enumerator SUBSET_TYPE_ID_RANGE
-
enumerator SUBSET_TYPE_ID_MOD
-
enumerator SUBSET_TYPE_ELEMENT_RANGE
-
enumerator SUBSET_TYPE_INVLIST_FRACTION
-
enumerator SUBSET_TYPE_INVLIST
-
enumerator SUBSET_TYPE_ID_RANGE
Public Functions
-
StopWordsInvertedLists(const InvertedLists *il, size_t maxsize)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
const InvertedLists *il0
-
size_t maxsize
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
enum subset_type_t
-
struct InvertedListsIOHook
- #include <InvertedListsIOHook.h>
Callbacks to handle other types of InvertedList objects.
The callbacks should be registered with add_callback before calling read_index or read_InvertedLists. The callbacks for OnDiskInvertedLists are registrered by default. The invlist type is identified by:
the key (a fourcc) at read time
the class name (as given by typeid.name) at write time
Subclassed by faiss::BlockInvertedListsIOHook, faiss::OnDiskInvertedListsIOHook
Public Functions
-
virtual void write(const InvertedLists *ils, IOWriter *f) const = 0
write the index to the IOWriter (including the fourcc)
-
virtual InvertedLists *read(IOReader *f, int io_flags) const = 0
called when the fourcc matches this class’s fourcc
-
virtual InvertedLists *read_ArrayInvertedLists(IOReader *f, int io_flags, size_t nlist, size_t code_size, const std::vector<size_t> &sizes) const
read from a ArrayInvertedLists into this invertedlist type. For this to work, the callback has to be enabled and the io_flag has to be set to IO_FLAG_SKIP_IVF_DATA | (16 upper bits of the fourcc)
(default implementation fails)
-
inline virtual ~InvertedListsIOHook()
Public Members
-
const std::string key
string version of the fourcc
-
const std::string classname
typeid.name
Public Static Functions
-
static void add_callback(InvertedListsIOHook*)
-
static void print_callbacks()
-
static InvertedListsIOHook *lookup(int h)
-
static InvertedListsIOHook *lookup_classname(const std::string &classname)
-
struct OnDiskOneList
Public Functions
-
OnDiskOneList()
Public Members
-
size_t size
-
size_t capacity
-
size_t offset
-
OnDiskOneList()
-
struct OnDiskInvertedLists : public faiss::InvertedLists
- #include <OnDiskInvertedLists.h>
On-disk storage of inverted lists.
The data is stored in a mmapped chunk of memory (base pointer ptr, size totsize). Each list is a range of memory that contains (object List) that contains:
uint8_t codes[capacity * code_size]
followed by idx_t ids[capacity]
in each of the arrays, the size <= capacity first elements are used, the rest is not initialized.
Addition and resize are supported by:
roundind up the capacity of the lists to a power of two
maintaining a list of empty slots, sorted by size.
resizing the mmapped block is adjusted as needed.
An OnDiskInvertedLists is compact if the size == capacity for all lists and there are no available slots.
Addition to the invlists is slow. For incremental add it is better to use a default ArrayInvertedLists object and convert it to an OnDisk with merge_from.
When it is known that a set of lists will be accessed, it is useful to call prefetch_lists, that launches a set of threads to read the lists in parallel.
Public Types
-
using List = OnDiskOneList
Public Functions
-
OnDiskInvertedLists(size_t nlist, size_t code_size, const char *filename)
are inverted lists mapped read-only
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
size_t merge_from_multiple(const InvertedLists **ils, int n_il, bool shift_ids = false, bool verbose = false)
-
size_t merge_from_1(const InvertedLists *il, bool verbose = false)
same as merge_from for a single invlist
-
void crop_invlists(size_t l0, size_t l1)
restrict the inverted lists to l0:l1 without touching the mmapped region
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
~OnDiskInvertedLists() override
-
void do_mmap()
-
void update_totsize(size_t new_totsize)
-
void resize_locked(size_t list_no, size_t new_size)
-
size_t allocate_slot(size_t capacity)
-
void free_slot(size_t offset, size_t capacity)
-
void set_all_lists_sizes(const size_t *sizes)
override all list sizes and make a packed storage
-
OnDiskInvertedLists()
-
bool is_empty(size_t list_no, void *inverted_list_context) const
-
virtual InvertedListsIterator *get_iterator(size_t list_no, void *inverted_list_context) const
get iterable for lists that use_iterator
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code, void *inverted_list_context = nullptr)
add one entry to an inverted list
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
size_t copy_subset_to(InvertedLists &other, subset_type_t subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
- Returns:
number of entries copied
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
std::string filename
-
size_t totsize
-
uint8_t *ptr
-
bool read_only
-
LockLevels *locks
-
OngoingPrefetch *pf
-
int prefetch_nthread
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
-
bool use_iterator
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
struct Slot
Public Functions
-
Slot(size_t offset, size_t capacity)
-
Slot()
Public Members
-
size_t offset
-
size_t capacity
-
Slot(size_t offset, size_t capacity)
-
struct OnDiskInvertedListsIOHook : public faiss::InvertedListsIOHook
Public Functions
-
OnDiskInvertedListsIOHook()
-
virtual void write(const InvertedLists *ils, IOWriter *f) const override
write the index to the IOWriter (including the fourcc)
-
virtual InvertedLists *read(IOReader *f, int io_flags) const override
called when the fourcc matches this class’s fourcc
-
virtual InvertedLists *read_ArrayInvertedLists(IOReader *f, int io_flags, size_t nlist, size_t code_size, const std::vector<size_t> &sizes) const override
read from a ArrayInvertedLists into this invertedlist type. For this to work, the callback has to be enabled and the io_flag has to be set to IO_FLAG_SKIP_IVF_DATA | (16 upper bits of the fourcc)
(default implementation fails)
Public Members
Public Static Functions
-
static void add_callback(InvertedListsIOHook*)
-
static void print_callbacks()
-
static InvertedListsIOHook *lookup(int h)
-
static InvertedListsIOHook *lookup_classname(const std::string &classname)
-
OnDiskInvertedListsIOHook()
-
struct MatrixStats
- #include <MatrixStats.h>
Reports some statistics on a dataset and comments on them.
It is a class rather than a function so that all stats can also be accessed from code
Public Functions
-
MatrixStats(size_t n, size_t d, const float *x)
-
void do_comment(const char *fmt, ...)
Public Members
-
std::string comments
-
size_t n = 0
-
size_t d = 0
-
size_t n_collision = 0
-
size_t n_valid = 0
-
size_t n0 = 0
-
double min_norm2 = HUGE_VALF
-
double max_norm2 = 0
-
uint64_t hash_value = 0
-
std::vector<PerDimStats> per_dim_stats
-
std::unordered_map<uint64_t, Occurrence> occurrences
-
char *buf
-
size_t nbuf
-
struct Occurrence
Public Members
-
size_t first
-
size_t count
-
size_t first
-
struct PerDimStats
Public Functions
-
void add(float x)
-
void compute_mean_std()
Public Members
-
size_t n = 0
counts of various special entries
-
size_t n_nan = 0
-
size_t n_inf = 0
-
size_t n0 = 0
-
float min = HUGE_VALF
to get min/max and stddev values
-
float max = -HUGE_VALF
-
double sum = 0
-
double sum2 = 0
-
size_t n_valid = 0
-
double mean = NAN
-
double stddev = NAN
-
void add(float x)
-
MatrixStats(size_t n, size_t d, const float *x)
-
struct IndexSplitVectors : public faiss::Index
- #include <MetaIndexes.h>
splits input vectors in segments and assigns each segment to a sub-index used to distribute a MultiIndexQuantizer
Public Functions
-
explicit IndexSplitVectors(idx_t d, bool threaded = false)
sum of dimensions seen so far
-
void add_sub_index(Index*)
-
void sync_with_sub_indexes()
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
~IndexSplitVectors() override
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
explicit IndexSplitVectors(idx_t d, bool threaded = false)
-
struct IndexRandom : public faiss::Index
- #include <MetaIndexes.h>
index that returns random results. used mainly for time benchmarks
Public Functions
-
explicit IndexRandom(idx_t d, idx_t ntotal = 0, int64_t seed = 1234, MetricType mt = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
n – number of vectors
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
~IndexRandom() override
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
n – number of vectors
x – input vectors, size n * d
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
k – number of nearest neighbours
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
k – number of extracted vectors
distances – output pairwise distances, size n*k
labels – output labels of the NNs, size n*k
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int64_t seed
-
int d
vector dimension
-
bool verbose
verbosity level
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
explicit IndexRandom(idx_t d, idx_t ntotal = 0, int64_t seed = 1234, MetricType mt = METRIC_L2)
-
template<class T, int A = 32>
struct AlignedTableTightAlloc Public Functions
-
inline AlignedTableTightAlloc()
-
inline explicit AlignedTableTightAlloc(size_t n)
-
inline size_t itemsize() const
-
inline void resize(size_t n)
-
inline void clear()
-
inline size_t size() const
-
inline size_t nbytes() const
-
inline T *get()
-
inline const T *get() const
-
inline T *data()
-
inline const T *data() const
-
inline T &operator[](size_t i)
-
inline T operator[](size_t i) const
-
inline ~AlignedTableTightAlloc()
-
inline AlignedTableTightAlloc<T, A> &operator=(const AlignedTableTightAlloc<T, A> &other)
-
inline AlignedTableTightAlloc(const AlignedTableTightAlloc<T, A> &other)
-
inline AlignedTableTightAlloc()
-
template<class T, int A = 32>
struct AlignedTable Public Functions
-
inline AlignedTable()
-
inline explicit AlignedTable(size_t n)
-
inline size_t itemsize() const
-
inline void resize(size_t n)
-
inline void clear()
-
inline size_t size() const
-
inline size_t nbytes() const
-
inline T *get()
-
inline const T *get() const
-
inline T *data()
-
inline const T *data() const
-
inline T &operator[](size_t i)
-
inline T operator[](size_t i) const
Public Static Functions
-
static inline size_t round_capacity(size_t n)
-
inline AlignedTable()
-
template<typename C, uint32_t NBUCKETS, uint32_t N>
struct HeapWithBuckets
-
template<uint32_t NBUCKETS, uint32_t N>
struct HeapWithBuckets<CMax<float, int>, NBUCKETS, N> Public Static Functions
- static inline void addn (const uint32_t n, const float *const __restrict distances, const uint32_t k, float *const __restrict bh_val, int32_t *const __restrict bh_ids)
- static inline void bs_addn (const uint32_t beam_size, const uint32_t n_per_beam, const float *const __restrict distances, const uint32_t k, float *const __restrict bh_val, int32_t *const __restrict bh_ids)
- static inline void addn (const uint32_t n, const float *const __restrict distances, const uint32_t k, float *const __restrict bh_val, int32_t *const __restrict bh_ids)
- static inline void bs_addn (const uint32_t beam_size, const uint32_t n_per_beam, const float *const __restrict distances, const uint32_t k, float *const __restrict bh_val, int32_t *const __restrict bh_ids)
-
template<typename C, uint32_t NBUCKETS, uint32_t N, typename HammingComputerT>
struct HeapWithBucketsForHamming32
-
template<uint32_t NBUCKETS, uint32_t N, typename HammingComputerT>
struct HeapWithBucketsForHamming32<CMax<int, int64_t>, NBUCKETS, N, HammingComputerT> Public Static Functions
- static inline void addn (const uint32_t n, const HammingComputerT &hc, const uint8_t *const __restrict binaryVectors, const uint32_t k, int *const __restrict bh_val, int64_t *const __restrict bh_ids)
- static inline void bs_addn (const uint32_t beam_size, const uint32_t n_per_beam, const HammingComputerT &hc, const uint8_t *const __restrict binary_vectors, const uint32_t k, int *const __restrict bh_val, int64_t *const __restrict bh_ids)
Public Static Attributes
-
static constexpr uint32_t NBUCKETS_8 = NBUCKETS / 8
-
template<typename C, uint32_t NBUCKETS, uint32_t N, typename HammingComputerT>
struct HeapWithBucketsForHamming16
-
template<uint32_t NBUCKETS, uint32_t N, typename HammingComputerT>
struct HeapWithBucketsForHamming16<CMax<int, int64_t>, NBUCKETS, N, HammingComputerT> Public Static Functions
- static inline void addn (const uint32_t n, const HammingComputerT &hc, const uint8_t *const __restrict binaryVectors, const uint32_t k, int *const __restrict bh_val, int64_t *const __restrict bh_ids)
- static inline void bs_addn (const uint32_t beam_size, const uint32_t n_per_beam, const HammingComputerT &hc, const uint8_t *const __restrict binary_vectors, const uint32_t k, int *const __restrict bh_val, int64_t *const __restrict bh_ids)
Public Static Attributes
-
static constexpr uint32_t NBUCKETS_16 = NBUCKETS / 16
-
template<MetricType mt>
struct VectorDistance Public Types
-
using C = typename std::conditional<is_similarity_metric(mt), CMin<float, int64_t>, CMax<float, int64_t>>::type
Public Functions
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
Public Members
-
size_t d
-
float metric_arg
Public Static Attributes
-
static constexpr bool is_similarity = is_similarity_metric(mt)
-
using C = typename std::conditional<is_similarity_metric(mt), CMin<float, int64_t>, CMax<float, int64_t>>::type
-
template<class HammingComputer>
struct HCounterState - #include <hamming-inl.h>
This class maintains a list of best distances seen so far.
Since the distances are in a limited range (0 to nbit), the object maintains one list per possible distance, and fills in only the n-first lists, such that the sum of sizes of the n lists is below k.
Public Functions
-
inline HCounterState(int *counters, int64_t *ids_per_dis, const uint8_t *x, int d, int k)
-
inline void update_counter(const uint8_t *y, size_t j)
Public Members
-
int *counters
-
int64_t *ids_per_dis
-
int thres
-
int count_lt
-
int count_eq
-
int k
-
inline HCounterState(int *counters, int64_t *ids_per_dis, const uint8_t *x, int d, int k)
-
struct BitstringWriter
Public Functions
-
inline BitstringWriter(uint8_t *code, size_t code_size)
-
inline void write(uint64_t x, int nbit)
Public Members
-
uint8_t *code
-
size_t code_size
-
size_t i
-
inline BitstringWriter(uint8_t *code, size_t code_size)
-
struct BitstringReader
Public Functions
-
inline BitstringReader(const uint8_t *code, size_t code_size)
-
inline uint64_t read(int nbit)
Public Members
-
const uint8_t *code
-
size_t code_size
-
size_t i
-
inline BitstringReader(const uint8_t *code, size_t code_size)
-
struct HammingComputer4
Public Functions
-
inline HammingComputer4()
-
inline HammingComputer4(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
-
inline HammingComputer4()
-
inline HammingComputer4(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
Public Members
-
uint32_t a0
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline HammingComputer4()
-
struct HammingComputer8
Public Functions
-
inline HammingComputer8()
-
inline HammingComputer8(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
-
inline HammingComputer8()
-
inline HammingComputer8(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
Public Members
-
uint64_t a0
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline HammingComputer8()
-
struct HammingComputer16
Public Functions
-
inline HammingComputer16()
-
inline HammingComputer16(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline HammingComputer16()
-
inline HammingComputer16(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline HammingComputer16()
-
struct HammingComputer20
Public Functions
-
inline HammingComputer20()
-
inline HammingComputer20(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline HammingComputer20()
-
inline HammingComputer20(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline HammingComputer20()
-
struct HammingComputer32
Public Functions
-
inline HammingComputer32()
-
inline HammingComputer32(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline HammingComputer32()
-
inline HammingComputer32(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline HammingComputer32()
-
struct HammingComputer64
Public Functions
-
inline HammingComputer64()
-
inline HammingComputer64(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline HammingComputer64()
-
inline HammingComputer64(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
uint64_t a0
-
uint64_t a1
-
uint64_t a2
-
uint64_t a3
-
uint64_t a4
-
uint64_t a5
-
uint64_t a6
-
uint64_t a7
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline HammingComputer64()
-
struct HammingComputerDefault
Subclassed by faiss::HammingComputer< CODE_SIZE >
Public Functions
-
inline HammingComputerDefault()
-
inline HammingComputerDefault(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8_2, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline int get_code_size() const
-
inline HammingComputerDefault()
-
inline HammingComputerDefault(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline int get_code_size() const
-
inline HammingComputerDefault()
-
struct GenHammingComputer8
Public Functions
-
inline GenHammingComputer8(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
-
inline GenHammingComputer8(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
Public Members
-
uint64_t a0
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline GenHammingComputer8(const uint8_t *a, int code_size)
-
struct GenHammingComputer16
Public Functions
-
inline GenHammingComputer16(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline GenHammingComputer16(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline GenHammingComputer16(const uint8_t *a8, int code_size)
-
struct GenHammingComputer32
Public Functions
-
inline GenHammingComputer32(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline GenHammingComputer32(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Static Functions
-
static inline constexpr int get_code_size()
-
static inline constexpr int get_code_size()
-
inline GenHammingComputer32(const uint8_t *a8, int code_size)
-
struct GenHammingComputerM8
-
template<int CODE_SIZE>
struct HammingComputer : public faiss::HammingComputerDefault Public Functions
-
inline HammingComputer(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a8_2, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
-
inline int hamming(const uint8_t *b8) const
-
inline int get_code_size() const
-
inline int get_code_size() const
-
inline HammingComputer(const uint8_t *a, int code_size)
-
template<typename C>
struct HeapArray - #include <Heap.h>
a template structure for a set of [min|max]-heaps it is tailored so that the actual data of the heaps can just live in compact arrays.
Public Functions
-
inline T *get_val(size_t key)
Return the list of values for a heap.
-
inline TI *get_ids(size_t key)
Correspponding identifiers.
-
void heapify()
prepare all the heaps before adding
-
void addn(size_t nj, const T *vin, TI j0 = 0, size_t i0 = 0, int64_t ni = -1)
add nj elements to heaps i0:i0+ni, with sequential ids
- Parameters:
nj – nb of elements to add to each heap
vin – elements to add, size ni * nj
j0 – add this to the ids that are added
i0 – first heap to update
ni – nb of elements to update (-1 = use nh)
-
void addn_with_ids(size_t nj, const T *vin, const TI *id_in = nullptr, int64_t id_stride = 0, size_t i0 = 0, int64_t ni = -1)
same as addn
- Parameters:
id_in – ids of the elements to add, size ni * nj
id_stride – stride for id_in
-
void addn_query_subset_with_ids(size_t nsubset, const TI *subset, size_t nj, const T *vin, const TI *id_in = nullptr, int64_t id_stride = 0)
same as addn_with_ids, but for just a subset of queries
- Parameters:
nsubset – number of query entries to update
subset – indexes of queries to update, in 0..nh-1, size nsubset
-
void reorder()
reorder all the heaps
-
inline T *get_val(size_t key)
-
template<typename T_, typename TI_>
struct CMax The C object gives the type T of the values of a key-value storage, the type of the keys, TI and the comparison that is done: CMax for a decreasing series and CMin for increasing series. In other words, for a given threshold threshold, an incoming value x is kept if
is true.C::cmp(threshold, x)
Public Static Functions
-
static inline T neutral()
Public Static Attributes
-
static const bool is_max = true
-
static inline T neutral()
-
template<typename T_, typename TI_>
struct CMin -
Public Static Functions
-
static inline T neutral()
Public Static Attributes
-
static const bool is_max = false
-
static inline T neutral()
-
struct PartitionStats
Public Functions
-
inline PartitionStats()
-
void reset()
Public Members
-
uint64_t bissect_cycles
-
uint64_t compress_cycles
-
inline PartitionStats()
-
struct RandomGenerator
- #include <random.h>
random generator that can be used in multithreaded contexts
Public Functions
-
int rand_int()
random positive integer
-
int64_t rand_int64()
random int64_t
-
int rand_int(int max)
generate random integer between 0 and max-1
-
float rand_float()
between 0 and 1
-
double rand_double()
-
explicit RandomGenerator(int64_t seed = 1234)
Public Members
-
std::mt19937 mt
-
int rand_int()
-
struct simd256bit
- #include <simdlib_avx2.h>
256-bit representation without interpretation as a vector
Simple wrapper around the AVX 256-bit registers
The objective is to separate the different interpretations of the same registers (as a vector of uint8, uint16 or uint32), to provide printing functions, and to give more readable names to the AVX intrinsics. It does not pretend to be exhausitve, functions are added as needed.
Subclassed by faiss::simd16uint16, faiss::simd16uint16, faiss::simd32uint8, faiss::simd32uint8, faiss::simd8float32, faiss::simd8float32, faiss::simd8uint32, faiss::simd8uint32
Public Functions
-
inline simd256bit()
-
inline explicit simd256bit(__m256i i)
-
inline explicit simd256bit(__m256 f)
-
inline explicit simd256bit(const void *x)
-
inline void clear()
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline void bin(char bits[257]) const
-
inline std::string bin() const
-
inline bool is_same_as(simd256bit other) const
-
inline simd256bit()
-
inline explicit simd256bit(const void *x)
-
inline void clear()
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline void bin(char bits[257]) const
-
inline std::string bin() const
-
inline bool is_same_as(simd256bit other) const
Public Members
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]
-
uint16_t u16[16]
-
uint32_t u32[8]
-
float f32[8]
-
union faiss::simd256bit::[anonymous] [anonymous]
-
inline simd256bit()
-
struct simd16uint16 : public faiss::simd256bit, public faiss::simd256bit
- #include <simdlib_avx2.h>
vector of 16 elements in uint16
Public Functions
-
inline simd16uint16()
-
inline explicit simd16uint16(__m256i i)
-
inline explicit simd16uint16(int x)
-
inline explicit simd16uint16(uint16_t x)
-
inline explicit simd16uint16(simd256bit x)
-
inline explicit simd16uint16(const uint16_t *x)
-
inline explicit simd16uint16(uint16_t u0, uint16_t u1, uint16_t u2, uint16_t u3, uint16_t u4, uint16_t u5, uint16_t u6, uint16_t u7, uint16_t u8, uint16_t u9, uint16_t u10, uint16_t u11, uint16_t u12, uint16_t u13, uint16_t u14, uint16_t u15)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint16_t x)
-
inline simd16uint16 operator*(const simd16uint16 &other) const
-
inline simd16uint16 operator>>(const int shift) const
-
inline simd16uint16 operator<<(const int shift) const
-
inline simd16uint16 operator+=(simd16uint16 other)
-
inline simd16uint16 operator-=(simd16uint16 other)
-
inline simd16uint16 operator+(simd16uint16 other) const
-
inline simd16uint16 operator-(simd16uint16 other) const
-
inline simd16uint16 operator&(simd256bit other) const
-
inline simd16uint16 operator|(simd256bit other) const
-
inline simd16uint16 operator^(simd256bit other) const
-
inline simd16uint16 operator~() const
-
inline uint16_t get_scalar_0() const
-
inline uint32_t ge_mask(simd16uint16 thresh) const
-
inline uint32_t le_mask(simd16uint16 thresh) const
-
inline uint32_t gt_mask(simd16uint16 thresh) const
-
inline bool all_gt(simd16uint16 thresh) const
-
inline uint16_t operator[](int i) const
-
inline void accu_min(simd16uint16 incoming)
-
inline void accu_max(simd16uint16 incoming)
-
inline simd16uint16()
-
inline explicit simd16uint16(int x)
-
inline explicit simd16uint16(uint16_t x)
-
inline explicit simd16uint16(const simd256bit &x)
-
inline explicit simd16uint16(const uint16_t *x)
-
inline explicit simd16uint16(uint16_t u0, uint16_t u1, uint16_t u2, uint16_t u3, uint16_t u4, uint16_t u5, uint16_t u6, uint16_t u7, uint16_t u8, uint16_t u9, uint16_t u10, uint16_t u11, uint16_t u12, uint16_t u13, uint16_t u14, uint16_t u15)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint16_t x)
-
inline simd16uint16 operator*(const simd16uint16 &other) const
-
inline simd16uint16 operator>>(const int shift) const
-
inline simd16uint16 operator<<(const int shift) const
-
inline simd16uint16 operator+=(const simd16uint16 &other)
-
inline simd16uint16 operator-=(const simd16uint16 &other)
-
inline simd16uint16 operator+(const simd16uint16 &other) const
-
inline simd16uint16 operator-(const simd16uint16 &other) const
-
inline simd16uint16 operator&(const simd256bit &other) const
-
inline simd16uint16 operator|(const simd256bit &other) const
-
inline simd16uint16 operator^(const simd256bit &other) const
-
inline simd16uint16 operator==(const simd16uint16 &other) const
-
inline simd16uint16 operator~() const
-
inline uint16_t get_scalar_0() const
-
inline uint32_t ge_mask(const simd16uint16 &thresh) const
-
inline uint32_t le_mask(const simd16uint16 &thresh) const
-
inline uint32_t gt_mask(const simd16uint16 &thresh) const
-
inline bool all_gt(const simd16uint16 &thresh) const
-
inline uint16_t operator[](int i) const
-
inline void accu_min(const simd16uint16 &incoming)
-
inline void accu_max(const simd16uint16 &incoming)
-
simd16uint16() = default
-
inline explicit simd16uint16(int x)
-
inline explicit simd16uint16(uint16_t x)
-
inline explicit simd16uint16(const uint16x8x2_t &v)
-
inline explicit simd16uint16(uint16_t u0, uint16_t u1, uint16_t u2, uint16_t u3, uint16_t u4, uint16_t u5, uint16_t u6, uint16_t u7, uint16_t u8, uint16_t u9, uint16_t u10, uint16_t u11, uint16_t u12, uint16_t u13, uint16_t u14, uint16_t u15)
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd16uint16(const T &x)
-
inline explicit simd16uint16(const uint16_t *x)
-
inline void clear()
-
inline void storeu(uint16_t *ptr) const
-
inline void loadu(const uint16_t *ptr)
-
inline void store(uint16_t *ptr) const
-
inline void bin(char bits[257]) const
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint16_t x)
-
inline simd16uint16 operator*(const simd16uint16 &other) const
-
inline simd16uint16 operator>>(const int shift) const
-
inline simd16uint16 operator<<(const int shift) const
-
inline simd16uint16 operator+=(const simd16uint16 &other)
-
inline simd16uint16 operator-=(const simd16uint16 &other)
-
inline simd16uint16 operator+(const simd16uint16 &other) const
-
inline simd16uint16 operator-(const simd16uint16 &other) const
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline simd16uint16 operator&(const T &other) const
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline simd16uint16 operator|(const T &other) const
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline simd16uint16 operator^(const T &other) const
-
inline simd16uint16 operator==(const simd16uint16 &other) const
-
inline simd16uint16 operator~() const
-
inline uint16_t get_scalar_0() const
-
inline uint32_t ge_mask(const simd16uint16 &thresh) const
-
inline uint32_t le_mask(const simd16uint16 &thresh) const
-
inline uint32_t gt_mask(const simd16uint16 &thresh) const
-
inline bool all_gt(const simd16uint16 &thresh) const
-
inline uint16_t operator[](int i) const
-
inline void accu_min(const simd16uint16 &incoming)
-
inline void accu_max(const simd16uint16 &incoming)
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline bool is_same_as(simd256bit other) const
Public Members
-
uint16x8x2_t data
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]
-
uint16_t u16[16]
-
uint32_t u32[8]
-
float f32[8]
-
union faiss::simd256bit::[anonymous] [anonymous]
Public Static Functions
-
template<typename F>
static inline simd16uint16 unary_func(const simd16uint16 &a, F &&f)
-
template<typename F>
static inline simd16uint16 binary_func(const simd16uint16 &a, const simd16uint16 &b, F &&f)
Friends
-
inline friend simd16uint16 operator==(const simd256bit lhs, const simd256bit rhs)
-
inline simd16uint16()
-
struct simd32uint8 : public faiss::simd256bit, public faiss::simd256bit
Public Functions
-
inline simd32uint8()
-
inline explicit simd32uint8(__m256i i)
-
inline explicit simd32uint8(int x)
-
inline explicit simd32uint8(uint8_t x)
-
inline explicit simd32uint8(simd256bit x)
-
inline explicit simd32uint8(const uint8_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint8_t x)
-
inline simd32uint8 operator&(simd256bit other) const
-
inline simd32uint8 operator+(simd32uint8 other) const
-
inline simd32uint8 lookup_2_lanes(simd32uint8 idx) const
-
inline simd16uint16 lane0_as_uint16() const
-
inline simd16uint16 lane1_as_uint16() const
-
inline simd32uint8 operator+=(simd32uint8 other)
-
inline uint8_t operator[](int i) const
-
inline simd32uint8()
-
inline explicit simd32uint8(int x)
-
inline explicit simd32uint8(uint8_t x)
-
inline explicit simd32uint8(const simd256bit &x)
-
inline explicit simd32uint8(const uint8_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint8_t x)
-
inline simd32uint8 operator&(const simd256bit &other) const
-
inline simd32uint8 operator+(const simd32uint8 &other) const
-
inline simd32uint8 lookup_2_lanes(const simd32uint8 &idx) const
-
inline simd32uint8 operator+=(const simd32uint8 &other)
-
inline uint8_t operator[](int i) const
-
simd32uint8() = default
-
inline explicit simd32uint8(int x)
-
inline explicit simd32uint8(uint8_t x)
-
inline explicit simd32uint8(const uint8x16x2_t &v)
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd32uint8(const T &x)
-
inline explicit simd32uint8(const uint8_t *x)
-
inline void clear()
-
inline void storeu(uint8_t *ptr) const
-
inline void loadu(const uint8_t *ptr)
-
inline void store(uint8_t *ptr) const
-
inline void bin(char bits[257]) const
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint8_t x)
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline simd32uint8 operator&(const T &other) const
-
inline simd32uint8 operator+(const simd32uint8 &other) const
-
inline simd32uint8 lookup_2_lanes(const simd32uint8 &idx) const
-
inline simd32uint8 operator+=(const simd32uint8 &other)
-
inline uint8_t operator[](int i) const
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline bool is_same_as(simd256bit other) const
Public Members
-
uint8x16x2_t data
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]
-
uint16_t u16[16]
-
uint32_t u32[8]
-
float f32[8]
-
union faiss::simd256bit::[anonymous] [anonymous]
Public Static Functions
-
template<uint8_t _0, uint8_t _1, uint8_t _2, uint8_t _3, uint8_t _4, uint8_t _5, uint8_t _6, uint8_t _7, uint8_t _8, uint8_t _9, uint8_t _10, uint8_t _11, uint8_t _12, uint8_t _13, uint8_t _14, uint8_t _15, uint8_t _16, uint8_t _17, uint8_t _18, uint8_t _19, uint8_t _20, uint8_t _21, uint8_t _22, uint8_t _23, uint8_t _24, uint8_t _25, uint8_t _26, uint8_t _27, uint8_t _28, uint8_t _29, uint8_t _30, uint8_t _31>
static inline simd32uint8 create()
-
template<uint8_t _0, uint8_t _1, uint8_t _2, uint8_t _3, uint8_t _4, uint8_t _5, uint8_t _6, uint8_t _7, uint8_t _8, uint8_t _9, uint8_t _10, uint8_t _11, uint8_t _12, uint8_t _13, uint8_t _14, uint8_t _15, uint8_t _16, uint8_t _17, uint8_t _18, uint8_t _19, uint8_t _20, uint8_t _21, uint8_t _22, uint8_t _23, uint8_t _24, uint8_t _25, uint8_t _26, uint8_t _27, uint8_t _28, uint8_t _29, uint8_t _30, uint8_t _31>
static inline simd32uint8 create()
-
template<typename F>
static inline simd32uint8 binary_func(const simd32uint8 &a, const simd32uint8 &b, F &&f)
-
template<uint8_t _0, uint8_t _1, uint8_t _2, uint8_t _3, uint8_t _4, uint8_t _5, uint8_t _6, uint8_t _7, uint8_t _8, uint8_t _9, uint8_t _10, uint8_t _11, uint8_t _12, uint8_t _13, uint8_t _14, uint8_t _15, uint8_t _16, uint8_t _17, uint8_t _18, uint8_t _19, uint8_t _20, uint8_t _21, uint8_t _22, uint8_t _23, uint8_t _24, uint8_t _25, uint8_t _26, uint8_t _27, uint8_t _28, uint8_t _29, uint8_t _30, uint8_t _31>
static inline simd32uint8 create()
-
inline simd32uint8()
-
struct simd8uint32 : public faiss::simd256bit, public faiss::simd256bit
- #include <simdlib_avx2.h>
vector of 8 unsigned 32-bit integers
Public Functions
-
inline simd8uint32()
-
inline explicit simd8uint32(__m256i i)
-
inline explicit simd8uint32(uint32_t x)
-
inline explicit simd8uint32(simd256bit x)
-
inline explicit simd8uint32(const uint8_t *x)
-
inline explicit simd8uint32(uint32_t u0, uint32_t u1, uint32_t u2, uint32_t u3, uint32_t u4, uint32_t u5, uint32_t u6, uint32_t u7)
-
inline simd8uint32 operator+(simd8uint32 other) const
-
inline simd8uint32 operator-(simd8uint32 other) const
-
inline simd8uint32 &operator+=(const simd8uint32 &other)
-
inline bool operator==(simd8uint32 other) const
-
inline bool operator!=(simd8uint32 other) const
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint32_t x)
-
inline simd8uint32 unzip() const
-
inline simd8uint32()
-
inline explicit simd8uint32(uint32_t x)
-
inline explicit simd8uint32(const simd256bit &x)
-
inline explicit simd8uint32(const uint32_t *x)
-
inline explicit simd8uint32(uint32_t u0, uint32_t u1, uint32_t u2, uint32_t u3, uint32_t u4, uint32_t u5, uint32_t u6, uint32_t u7)
-
inline simd8uint32 operator+(simd8uint32 other) const
-
inline simd8uint32 operator-(simd8uint32 other) const
-
inline simd8uint32 &operator+=(const simd8uint32 &other)
-
inline bool operator==(simd8uint32 other) const
-
inline bool operator!=(simd8uint32 other) const
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint32_t x)
-
inline simd8uint32 unzip() const
-
simd8uint32() = default
-
inline explicit simd8uint32(uint32_t x)
-
inline explicit simd8uint32(const uint32x4x2_t &v)
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd8uint32(const T &x)
-
inline explicit simd8uint32(const uint8_t *x)
-
inline explicit simd8uint32(uint32_t u0, uint32_t u1, uint32_t u2, uint32_t u3, uint32_t u4, uint32_t u5, uint32_t u6, uint32_t u7)
-
inline simd8uint32 operator+(simd8uint32 other) const
-
inline simd8uint32 operator-(simd8uint32 other) const
-
inline simd8uint32 &operator+=(const simd8uint32 &other)
-
inline simd8uint32 operator==(simd8uint32 other) const
-
inline simd8uint32 operator~() const
-
inline simd8uint32 operator!=(simd8uint32 other) const
-
inline void clear()
-
inline void storeu(uint32_t *ptr) const
-
inline void loadu(const uint32_t *ptr)
-
inline void store(uint32_t *ptr) const
-
inline void bin(char bits[257]) const
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint32_t x)
-
inline simd8uint32 unzip() const
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline bool is_same_as(simd256bit other) const
Public Members
-
uint32x4x2_t data
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]
-
uint16_t u16[16]
-
uint32_t u32[8]
-
float f32[8]
-
union faiss::simd256bit::[anonymous] [anonymous]
-
inline simd8uint32()
-
struct simd8float32 : public faiss::simd256bit, public faiss::simd256bit
Public Functions
-
inline simd8float32()
-
inline explicit simd8float32(simd256bit x)
-
inline explicit simd8float32(__m256 x)
-
inline explicit simd8float32(float x)
-
inline explicit simd8float32(const float *x)
-
inline explicit simd8float32(float f0, float f1, float f2, float f3, float f4, float f5, float f6, float f7)
-
inline simd8float32 operator*(simd8float32 other) const
-
inline simd8float32 operator+(simd8float32 other) const
-
inline simd8float32 operator-(simd8float32 other) const
-
inline simd8float32 &operator+=(const simd8float32 &other)
-
inline bool operator==(simd8float32 other) const
-
inline bool operator!=(simd8float32 other) const
-
inline std::string tostring() const
-
inline simd8float32()
-
inline explicit simd8float32(const simd256bit &x)
-
inline explicit simd8float32(float x)
-
inline explicit simd8float32(const float *x)
-
inline void set1(float x)
-
inline explicit simd8float32(float f0, float f1, float f2, float f3, float f4, float f5, float f6, float f7)
-
inline simd8float32 operator*(const simd8float32 &other) const
-
inline simd8float32 operator+(const simd8float32 &other) const
-
inline simd8float32 operator-(const simd8float32 &other) const
-
inline simd8float32 &operator+=(const simd8float32 &other)
-
inline bool operator==(simd8float32 other) const
-
inline bool operator!=(simd8float32 other) const
-
inline std::string tostring() const
-
simd8float32() = default
-
inline explicit simd8float32(float x)
-
inline explicit simd8float32(const float32x4x2_t &v)
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd8float32(const T &x)
-
inline explicit simd8float32(const float *x)
-
inline explicit simd8float32(float f0, float f1, float f2, float f3, float f4, float f5, float f6, float f7)
-
inline void clear()
-
inline void storeu(float *ptr) const
-
inline void loadu(const float *ptr)
-
inline void store(float *ptr) const
-
inline void bin(char bits[257]) const
-
inline simd8float32 operator*(const simd8float32 &other) const
-
inline simd8float32 operator+(const simd8float32 &other) const
-
inline simd8float32 operator-(const simd8float32 &other) const
-
inline simd8float32 &operator+=(const simd8float32 &other)
-
inline simd8uint32 operator==(simd8float32 other) const
-
inline simd8uint32 operator!=(simd8float32 other) const
-
inline std::string tostring() const
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline bool is_same_as(simd256bit other) const
Public Members
-
float32x4x2_t data
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]
-
uint16_t u16[16]
-
uint32_t u32[8]
-
float f32[8]
-
union faiss::simd256bit::[anonymous] [anonymous]
Public Static Functions
-
template<typename F>
static inline simd8float32 binary_func(const simd8float32 &a, const simd8float32 &b, F &&f)
-
inline simd8float32()
-
struct simd512bit
- #include <simdlib_avx512.h>
512-bit representation without interpretation as a vector
Simple wrapper around the AVX 512-bit registers
The objective is to separate the different interpretations of the same registers (as a vector of uint8, uint16 or uint32), to provide printing functions, and to give more readable names to the AVX intrinsics. It does not pretend to be exhausitve, functions are added as needed.
Subclassed by faiss::simd32uint16, faiss::simd64uint8
Public Functions
-
inline simd512bit()
-
inline explicit simd512bit(__m512i i)
-
inline explicit simd512bit(__m512 f)
-
inline explicit simd512bit(const void *x)
-
inline explicit simd512bit(simd256bit lo)
-
inline explicit simd512bit(simd256bit lo, simd256bit hi)
-
inline void clear()
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline void bin(char bits[513]) const
-
inline std::string bin() const
-
inline simd512bit()
-
struct simd32uint16 : public faiss::simd512bit
- #include <simdlib_avx512.h>
vector of 32 elements in uint16
Public Functions
-
inline simd32uint16()
-
inline explicit simd32uint16(__m512i i)
-
inline explicit simd32uint16(int x)
-
inline explicit simd32uint16(uint16_t x)
-
inline explicit simd32uint16(simd512bit x)
-
inline explicit simd32uint16(const uint16_t *x)
-
inline explicit simd32uint16(simd256bit lo)
-
inline explicit simd32uint16(simd256bit lo, simd256bit hi)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint16_t x)
-
inline simd32uint16 operator*(const simd32uint16 &other) const
-
inline simd32uint16 operator>>(const int shift) const
-
inline simd32uint16 operator<<(const int shift) const
-
inline simd32uint16 operator+=(simd32uint16 other)
-
inline simd32uint16 operator-=(simd32uint16 other)
-
inline simd32uint16 operator+(simd32uint16 other) const
-
inline simd32uint16 operator-(simd32uint16 other) const
-
inline simd32uint16 operator&(simd512bit other) const
-
inline simd32uint16 operator|(simd512bit other) const
-
inline simd32uint16 operator^(simd512bit other) const
-
inline simd32uint16 operator~() const
-
inline simd16uint16 low() const
-
inline simd16uint16 high() const
-
inline uint16_t operator[](int i) const
-
inline void accu_min(simd32uint16 incoming)
-
inline void accu_max(simd32uint16 incoming)
-
inline void clear()
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline void bin(char bits[513]) const
-
inline std::string bin() const
-
inline simd32uint16()
-
struct simd64uint8 : public faiss::simd512bit
Public Functions
-
inline simd64uint8()
-
inline explicit simd64uint8(__m512i i)
-
inline explicit simd64uint8(int x)
-
inline explicit simd64uint8(uint8_t x)
-
inline explicit simd64uint8(simd256bit lo)
-
inline explicit simd64uint8(simd256bit lo, simd256bit hi)
-
inline explicit simd64uint8(simd512bit x)
-
inline explicit simd64uint8(const uint8_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint8_t x)
-
inline simd64uint8 operator&(simd512bit other) const
-
inline simd64uint8 operator+(simd64uint8 other) const
-
inline simd64uint8 lookup_4_lanes(simd64uint8 idx) const
-
inline simd32uint16 lane0_as_uint16() const
-
inline simd32uint16 lane1_as_uint16() const
-
inline simd64uint8 operator+=(simd64uint8 other)
-
inline uint8_t operator[](int i) const
-
inline void clear()
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline void bin(char bits[513]) const
-
inline std::string bin() const
-
inline simd64uint8()
-
template<typename T>
struct CombinerRangeKNN - #include <utils.h>
This class is used to combine range and knn search results in contrib.exhaustive_search.range_search_gpu
Public Functions
-
inline CombinerRangeKNN(int64_t nq, size_t k, T r2, bool keep_max)
whether to keep max values instead of min.
-
void compute_sizes(int64_t *L_res)
size nq + 1
-
void write_result(T *D_res, int64_t *I_res)
Phase 2: caller allocates D_res and I_res (size L_res[nq]) Phase 3: fill in D_res and I_res
Public Members
-
int64_t nq
-
size_t k
nb of queries
-
T r2
number of neighbors for the knn search part
-
bool keep_max
range search radius
-
const int64_t *I = nullptr
Knn search results.
-
const T *D = nullptr
size nq * k
-
const bool *mask = nullptr
size nq * k
optional: range search results (ignored if mask is NULL)
-
const int64_t *lim_remain = nullptr
mask for where knn results are valid, size nq
-
const T *D_remain = nullptr
size nrange + 1
-
const int64_t *I_remain = nullptr
size lim_remain[nrange]
-
const int64_t *L_res = nullptr
size lim_remain[nrange]
-
inline CombinerRangeKNN(int64_t nq, size_t k, T r2, bool keep_max)
-
struct CodeSet
Public Functions
-
inline explicit CodeSet(size_t d)
-
void insert(size_t n, const uint8_t *codes, bool *inserted)
-
inline explicit CodeSet(size_t d)
-
class WorkerThread
Public Functions
-
WorkerThread()
-
~WorkerThread()
Stops and waits for the worker thread to exit, flushing all pending lambdas
-
void stop()
Request that the worker thread stop itself.
-
void waitForThreadExit()
Blocking waits in the current thread for the worker thread to stop
Private Functions
-
void startThread()
-
void threadMain()
-
void threadLoop()
-
WorkerThread()
-
struct VectorTransform
- #include <VectorTransform.h>
Any transformation applied on a set of vectors
Subclassed by faiss::CenteringTransform, faiss::ITQTransform, faiss::LinearTransform, faiss::NormalizationTransform, faiss::RemapDimensionsTransform
Public Functions
-
inline explicit VectorTransform(int d_in = 0, int d_out = 0)
! output dimension
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const = 0
apply the transformation and return the result in a provided matrix
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
xt – output vectors, size n * d_out
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const
reverse transformation. May not be implemented or may return approximate result
-
virtual void check_identical(const VectorTransform &other) const = 0
-
inline virtual ~VectorTransform()
Public Members
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
inline explicit VectorTransform(int d_in = 0, int d_out = 0)
-
struct LinearTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
Generic linear transformation, with bias term applied on output y = A * x + b
Subclassed by faiss::ITQMatrix, faiss::OPQMatrix, faiss::PCAMatrix, faiss::RandomRotationMatrix
Public Functions
-
explicit LinearTransform(int d_in = 0, int d_out = 0, bool have_bias = false)
both d_in > d_out and d_out < d_in are supported
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
virtual void check_identical(const VectorTransform &other) const override
-
inline ~LinearTransform() override
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit LinearTransform(int d_in = 0, int d_out = 0, bool have_bias = false)
-
struct RandomRotationMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Randomly rotate a set of vectors.
Public Functions
-
inline RandomRotationMatrix(int d_in, int d_out)
both d_in > d_out and d_out < d_in are supported
-
void init(int seed)
must be called before the transform is used
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
inline RandomRotationMatrix()
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
virtual void check_identical(const VectorTransform &other) const override
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
inline RandomRotationMatrix(int d_in, int d_out)
-
struct PCAMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Applies a principal component analysis on a set of vectors, with optionally whitening and random rotation.
Public Functions
-
explicit PCAMatrix(int d_in = 0, int d_out = 0, float eigen_power = 0, bool random_rotation = false)
-
virtual void train(idx_t n, const float *x) override
train on n vectors. If n < d_in then the eigenvector matrix will be completed with 0s
-
void copy_from(const PCAMatrix &other)
copy pre-trained PCA matrix
-
void prepare_Ab()
called after mean, PCAMat and eigenvalues are computed
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
virtual void check_identical(const VectorTransform &other) const override
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
float eigen_power
after transformation the components are multiplied by eigenvalues^eigen_power
=0: no whitening =-0.5: full whitening
-
float epsilon
value added to eigenvalues to avoid division by 0 when whitening
-
bool random_rotation
random rotation after PCA
-
size_t max_points_per_d
ratio between # training vectors and dimension
-
int balanced_bins
try to distribute output eigenvectors in this many bins
-
std::vector<float> mean
Mean, size d_in.
-
std::vector<float> eigenvalues
eigenvalues of covariance matrix (= squared singular values)
-
std::vector<float> PCAMat
PCA matrix, size d_in * d_in.
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit PCAMatrix(int d_in = 0, int d_out = 0, float eigen_power = 0, bool random_rotation = false)
-
struct ITQMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
ITQ implementation from
Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval,
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, Florent Perronnin, PAMI’12.
Public Functions
-
explicit ITQMatrix(int d = 0)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
virtual void check_identical(const VectorTransform &other) const override
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
int max_iter
-
int seed
-
std::vector<double> init_rotation
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit ITQMatrix(int d = 0)
-
struct ITQTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
The full ITQ transform, including normalizations and PCA transformation
Public Functions
-
explicit ITQTransform(int d_in = 0, int d_out = 0, bool do_pca = false)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
apply the transformation and return the result in a provided matrix
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
xt – output vectors, size n * d_out
-
virtual void check_identical(const VectorTransform &other) const override
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const
reverse transformation. May not be implemented or may return approximate result
Public Members
-
std::vector<float> mean
-
bool do_pca
-
ITQMatrix itq
-
int max_train_per_dim
max training points per dimension
-
LinearTransform pca_then_itq
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit ITQTransform(int d_in = 0, int d_out = 0, bool do_pca = false)
-
struct OPQMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Applies a rotation to align the dimensions with a PQ to minimize the reconstruction error. Can be used before an IndexPQ or an IndexIVFPQ. The method is the non-parametric version described in:
“Optimized Product Quantization for Approximate Nearest Neighbor Search” Tiezheng Ge, Kaiming He, Qifa Ke, Jian Sun, CVPR’13
Public Functions
-
explicit OPQMatrix(int d = 0, int M = 1, int d2 = -1)
if d2 != -1, output vectors of this dimension
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
virtual void check_identical(const VectorTransform &other) const override
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
int M
nb of subquantizers
-
int niter = 50
Number of outer training iterations.
-
int niter_pq = 4
Number of training iterations for the PQ.
-
int niter_pq_0 = 40
same, for the first outer iteration
-
size_t max_train_points = 256 * 256
if there are too many training points, resample
-
bool verbose = false
-
ProductQuantizer *pq = nullptr
if non-NULL, use this product quantizer for training should be constructed with (d_out, M, _)
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit OPQMatrix(int d = 0, int M = 1, int d2 = -1)
-
struct RemapDimensionsTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
remap dimensions for intput vectors, possibly inserting 0s strictly speaking this is also a linear transform but we don’t want to compute it with matrix multiplies
Public Functions
-
RemapDimensionsTransform(int d_in, int d_out, const int *map)
-
RemapDimensionsTransform(int d_in, int d_out, bool uniform = true)
remap input to output, skipping or inserting dimensions as needed if uniform: distribute dimensions uniformly otherwise just take the d_out first ones.
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
apply the transformation and return the result in a provided matrix
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
xt – output vectors, size n * d_out
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
reverse transform correct only when the mapping is a permutation
-
inline RemapDimensionsTransform()
-
virtual void check_identical(const VectorTransform &other) const override
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
std::vector<int> map
map from output dimension to input, size d_out -1 -> set output to 0
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
RemapDimensionsTransform(int d_in, int d_out, const int *map)
-
struct NormalizationTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
per-vector normalization
Public Functions
-
explicit NormalizationTransform(int d, float norm = 2.0)
-
NormalizationTransform()
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
apply the transformation and return the result in a provided matrix
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
xt – output vectors, size n * d_out
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
Identity transform since norm is not revertible.
-
virtual void check_identical(const VectorTransform &other) const override
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
float norm
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit NormalizationTransform(int d, float norm = 2.0)
-
struct CenteringTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
Subtract the mean of each component from the vectors.
Public Functions
-
explicit CenteringTransform(int d = 0)
-
virtual void train(idx_t n, const float *x) override
train on n vectors.
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
subtract the mean
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
add the mean
-
virtual void check_identical(const VectorTransform &other) const override
-
float *apply(idx_t n, const float *x) const
apply the transformation and return the result in an allocated pointer
- Parameters:
n – number of vectors to transform
x – input vectors, size n * d_in
- Returns:
output vectors, size n * d_out
Public Members
-
std::vector<float> mean
Mean, size d_in = d_out.
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit CenteringTransform(int d = 0)
-
namespace cppcontrib
-
template<intptr_t DIM, intptr_t COARSE_SIZE, intptr_t FINE_SIZE, intptr_t COARSE_BITS = 8, intptr_t FINE_BITS = 8>
struct Index2LevelDecoder - #include <Level2-inl.h>
Public Types
-
using coarse_storage_type = typename detail::CoarseBitType<COARSE_BITS>::bit_type
Public Static Functions
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqCoarseCentroids2, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqCoarseCentroids2, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqCoarseCentroids2, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
Public Static Attributes
-
static constexpr intptr_t dim = DIM
-
static constexpr intptr_t coarseSize = COARSE_SIZE
-
static constexpr intptr_t fineSize = FINE_SIZE
-
static constexpr intptr_t coarseBits = COARSE_BITS
-
static constexpr intptr_t fineBits = FINE_BITS
-
static constexpr intptr_t COARSE_TABLE_BYTES = (1 << COARSE_BITS)
-
using coarse_storage_type = typename detail::CoarseBitType<COARSE_BITS>::bit_type
-
template<typename SubIndexT>
struct IndexMinMaxDecoder Public Static Functions
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void store (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqCoarseCentroids2, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
Public Static Attributes
-
static constexpr intptr_t dim = SubIndexT::dim
-
template<typename SubIndexT>
struct IndexMinMaxFP16Decoder Public Static Functions
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void store (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqCoarseCentroids2, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum, float &minvAccum)
Public Static Attributes
-
static constexpr intptr_t dim = SubIndexT::dim
-
template<intptr_t DIM, intptr_t FINE_SIZE, intptr_t FINE_BITS = 8>
struct IndexPQDecoder - #include <PQ-inl.h>
Public Static Functions
- static inline void store (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void store (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void store (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, const float *const __restrict pqFineCentroids2, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code0, const float weight0, const uint8_t *const __restrict code1, const float weight1, const uint8_t *const __restrict code2, const float weight2, float *const __restrict outputAccum)
-
namespace detail
Typedefs
-
template<intptr_t DIM, intptr_t CODE_SIZE, intptr_t CODE_BITS, intptr_t CPOS>
using UintReader = typename UintReaderImplType<DIM / CODE_SIZE, CODE_BITS, CPOS>::reader_type
-
template<intptr_t N_ELEMENTS, intptr_t CODE_BITS, intptr_t CPOS>
using UintReaderRaw = typename UintReaderImplType<N_ELEMENTS, CODE_BITS, CPOS>::reader_type
-
template<int COARSE_BITS>
struct CoarseBitType
-
template<>
struct CoarseBitType<8> Public Types
-
using bit_type = uint8_t
-
using bit_type = uint8_t
-
template<>
struct CoarseBitType<16> Public Types
-
using bit_type = uint16_t
-
using bit_type = uint16_t
-
template<intptr_t DIM, intptr_t CODE_SIZE, intptr_t CODE_BITS, intptr_t CPOS>
-
template<intptr_t DIM, intptr_t COARSE_SIZE, intptr_t FINE_SIZE, intptr_t COARSE_BITS = 8, intptr_t FINE_BITS = 8>
-
namespace detail
-
namespace simdlib
Typedefs
Functions
-
static inline uint8x16x2_t reinterpret_u8(const uint8x16x2_t &v)
-
static inline uint8x16x2_t reinterpret_u8(const uint16x8x2_t &v)
-
static inline uint8x16x2_t reinterpret_u8(const uint32x4x2_t &v)
-
static inline uint8x16x2_t reinterpret_u8(const float32x4x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const uint8x16x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const uint16x8x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const uint32x4x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const float32x4x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const uint8x16x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const uint16x8x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const uint32x4x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const float32x4x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const uint8x16x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const uint16x8x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const uint32x4x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const float32x4x2_t &v)
-
static inline ::uint16x8_t vdupq_n_u16(std::uint16_t v)
-
static inline ::uint8x16_t vdupq_n_u8(std::uint8_t v)
-
static inline void bin(const char (&bytes)[32], char bits[257])
-
template<typename T, size_t N, typename S>
static inline void bin(const S &simd, char bits[257])
-
template<typename T, size_t N, typename S>
static inline std::string elements_to_string(const char *fmt, const S &simd)
-
template<typename T>
static inline unary_func_impl<remove_cv_ref_t<T>, remove_cv_ref_t<T>> unary_func(const T &a)
-
template<typename T, typename U>
static inline unary_func_impl<remove_cv_ref_t<T>, remove_cv_ref_t<U>> unary_func(const U &a)
-
template<typename T>
static inline binary_func_impl<remove_cv_ref_t<T>, remove_cv_ref_t<T>> binary_func(const T &a, const T &b)
-
template<typename T, typename U>
static inline binary_func_impl<remove_cv_ref_t<T>, remove_cv_ref_t<U>> binary_func(const U &a, const U &b)
-
static inline uint16_t vmovmask_u8(const uint8x16_t &v)
-
template<uint16x8_t (*F)(uint16x8_t, uint16x8_t)>
static inline uint32_t cmp_xe32(const uint16x8x2_t &d0, const uint16x8x2_t &d1, const uint16x8x2_t &thr)
-
template<std::uint8_t Shift>
static inline uint16x8_t vshlq(uint16x8_t vec)
-
template<std::uint8_t Shift>
static inline uint16x8_t vshrq(uint16x8_t vec)
-
template<typename T, typename U = decltype(reinterpret_u8(std::declval<T>().data))>
struct is_simd256bit : public std::is_same<decltype(reinterpret_u8(std::declval<T>().data)), uint8x16x2_t>
-
template<typename D, typename T>
struct set1_impl Public Functions
-
template<remove_cv_ref_t<decltype(std::declval<D>().val[0])> (*F)(T)>
inline void call()
-
template<remove_cv_ref_t<decltype(std::declval<D>().val[0])> (*F)(T)>
-
template<typename T, typename U>
struct unary_func_impl Public Types
-
using Telem = remove_cv_ref_t<decltype(std::declval<T>().val[0])>
-
using Uelem = remove_cv_ref_t<decltype(std::declval<U>().val[0])>
Public Members
-
const U &a
-
using Telem = remove_cv_ref_t<decltype(std::declval<T>().val[0])>
-
template<typename T, typename U>
struct binary_func_impl Public Types
-
using Telem = remove_cv_ref_t<decltype(std::declval<T>().val[0])>
-
using Uelem = remove_cv_ref_t<decltype(std::declval<U>().val[0])>
-
using Telem = remove_cv_ref_t<decltype(std::declval<T>().val[0])>
-
static inline uint8x16x2_t reinterpret_u8(const uint8x16x2_t &v)
-
namespace simdlib
-
namespace gpu
Enums
-
enum class DistanceDataType
Values:
-
enumerator F32
-
enumerator F16
-
enumerator F32
-
enum class IndicesDataType
Values:
-
enumerator I64
-
enumerator I32
-
enumerator I64
-
enum IndicesOptions
How user vector index data is stored on the GPU.
Values:
-
enumerator INDICES_CPU
The user indices are only stored on the CPU; the GPU returns (inverted list, offset) to the CPU which is then translated to the real user index.
-
enumerator INDICES_IVF
The indices are not stored at all, on either the CPU or GPU. Only (inverted list, offset) is returned to the user as the index.
-
enumerator INDICES_32_BIT
Indices are stored as 32 bit integers on the GPU, but returned as 64 bit integers
-
enumerator INDICES_64_BIT
Indices are stored as 64 bit integers on the GPU.
-
enumerator INDICES_CPU
-
enum AllocType
Values:
-
enumerator Other
Unknown allocation type or miscellaneous (not currently categorized)
-
enumerator FlatData
Primary data storage for GpuIndexFlat (the raw matrix of vectors and vector norms if needed)
-
enumerator IVFLists
Primary data storage for GpuIndexIVF* (the storage for each individual IVF list)
-
enumerator Quantizer
Quantizer (PQ, SQ) dictionary information.
-
enumerator QuantizerPrecomputedCodes
For GpuIndexIVFPQ, “precomputed codes” for more efficient PQ lookup require the use of possibly large tables. These are marked separately from Quantizer as these can frequently be 100s - 1000s of MiB in size
-
enumerator TemporaryMemoryBuffer
StandardGpuResources implementation specific types When using StandardGpuResources, temporary memory allocations (MemorySpace::Temporary) come out of a stack region of memory that is allocated up front for each gpu (e.g., 1.5 GiB upon initialization). This allocation by StandardGpuResources is marked with this AllocType.
-
enumerator TemporaryMemoryOverflow
When using StandardGpuResources, any MemorySpace::Temporary allocations that cannot be satisfied within the TemporaryMemoryBuffer region fall back to calling cudaMalloc which are sized to just the request at hand. These “overflow” temporary allocations are marked with this AllocType.
-
enumerator Other
-
enum MemorySpace
Memory regions accessible to the GPU.
Values:
-
enumerator Temporary
Temporary device memory (guaranteed to no longer be used upon exit of a top-level index call, and where the streams using it have completed GPU work). Typically backed by Device memory (cudaMalloc/cudaFree).
-
enumerator Device
Managed using cudaMalloc/cudaFree (typical GPU device memory)
-
enumerator Unified
Managed using cudaMallocManaged/cudaFree (typical Unified CPU/GPU memory)
-
enumerator Temporary
Functions
-
faiss::Index *index_gpu_to_cpu(const faiss::Index *gpu_index)
converts any GPU index inside gpu_index to a CPU index
-
faiss::Index *index_cpu_to_gpu(GpuResourcesProvider *provider, int device, const faiss::Index *index, const GpuClonerOptions *options = nullptr)
converts any CPU index that can be converted to GPU
-
faiss::Index *index_cpu_to_gpu_multiple(std::vector<GpuResourcesProvider*> &provider, std::vector<int> &devices, const faiss::Index *index, const GpuMultipleClonerOptions *options = nullptr)
-
faiss::IndexBinary *index_binary_gpu_to_cpu(const faiss::IndexBinary *gpu_index)
-
faiss::IndexBinary *index_binary_cpu_to_gpu(GpuResourcesProvider *provider, int device, const faiss::IndexBinary *index, const GpuClonerOptions *options = nullptr)
converts any CPU index that can be converted to GPU
-
faiss::IndexBinary *index_binary_cpu_to_gpu_multiple(std::vector<GpuResourcesProvider*> &provider, std::vector<int> &devices, const faiss::IndexBinary *index, const GpuMultipleClonerOptions *options = nullptr)
-
bool should_use_raft(GpuDistanceParams args)
A function that determines whether RAFT should be used based on various conditions (such as unsupported architecture)
-
void bfKnn(GpuResourcesProvider *resources, const GpuDistanceParams &args)
A wrapper for gpu/impl/Distance.cuh to expose direct brute-force k-nearest neighbor searches on an externally-provided region of memory (e.g., from a pytorch tensor). The data (vectors, queries, outDistances, outIndices) can be resident on the GPU or the CPU, but all calculations are performed on the GPU. If the result buffers are on the CPU, results will be copied back when done.
All GPU computation is performed on the current CUDA device, and ordered with respect to resources->getDefaultStreamCurrentDevice().
For each vector in
queries, searches all ofvectorsto find its k nearest neighbors with respect to the given metric
-
void bfKnn_tiling(GpuResourcesProvider *resources, const GpuDistanceParams &args, size_t vectorsMemoryLimit, size_t queriesMemoryLimit)
-
void bruteForceKnn(GpuResourcesProvider *resources, faiss::MetricType metric, const float *vectors, bool vectorsRowMajor, idx_t numVectors, const float *queries, bool queriesRowMajor, idx_t numQueries, int dims, int k, float *outDistances, idx_t *outIndices)
Deprecated legacy implementation.
-
bool should_use_raft(GpuIndexConfig config_)
A centralized function that determines whether RAFT should be used based on various conditions (such as unsupported architecture)
-
GpuIndex *tryCastGpuIndex(faiss::Index *index)
If the given index is a GPU index, this returns the index instance.
-
bool isGpuIndexImplemented(faiss::Index *index)
Does the given CPU index instance have a corresponding GPU implementation?
-
std::string memorySpaceToString(MemorySpace s)
Convert a MemorySpace to string.
-
AllocInfo makeDevAlloc(AllocType at, cudaStream_t st)
Create an AllocInfo for the current device with MemorySpace::Device.
-
AllocInfo makeTempAlloc(AllocType at, cudaStream_t st)
Create an AllocInfo for the current device with MemorySpace::Temporary.
-
AllocInfo makeSpaceAlloc(AllocType at, MemorySpace sp, cudaStream_t st)
Create an AllocInfo for the current device.
-
int getMaxKSelection()
A collection of various utility functions for index implementation.
Returns the maximum k-selection value supported based on the CUDA SDK that we were compiled with. .cu files can use DeviceDefs.cuh, but this is for non-CUDA files
-
void validateKSelect(int k)
-
void validateNProbe(size_t nprobe)
-
std::vector<uint8_t> unpackNonInterleaved(std::vector<uint8_t> data, int numVecs, int dims, int bitsPerCode)
-
std::vector<uint8_t> unpackInterleaved(std::vector<uint8_t> data, int numVecs, int dims, int bitsPerCode)
-
std::vector<uint8_t> packNonInterleaved(std::vector<uint8_t> data, int numVecs, int dims, int bitsPerCode)
-
std::vector<uint8_t> packInterleaved(std::vector<uint8_t> data, int numVecs, int dims, int bitsPerCode)
-
void ivfOffsetToUserIndex(idx_t *indices, idx_t numLists, idx_t queries, int k, const std::vector<std::vector<idx_t>> &listOffsetToUserIndex)
Utility function to translate (list id, offset) to a user index on the CPU. In a cpp in order to use OpenMP.
-
void newTestSeed()
Generates and displays a new seed for the test.
-
void setTestSeed(long seed)
Uses an explicit seed for the test.
-
float relativeError(float a, float b)
Returns the relative error in difference between a and b (|a - b| / (0.5 * (|a| + |b|))
-
int randVal(int a, int b)
Generates a random integer in the range [a, b].
-
bool randBool()
Generates a random bool.
-
template<typename T>
T randSelect(std::initializer_list<T> vals) Select a random value from the given list of values provided as an initializer_list
-
std::vector<float> randVecs(size_t num, size_t dim)
Generates a collection of random vectors in the range [0, 1].
-
std::vector<unsigned char> randBinaryVecs(size_t num, size_t dim)
Generates a collection of random bit vectors.
-
void compareIndices(const std::vector<float> &queryVecs, faiss::Index &refIndex, faiss::Index &testIndex, int numQuery, int dim, int k, const std::string &configMsg, float maxRelativeError = 6e-5f, float pctMaxDiff1 = 0.1f, float pctMaxDiffN = 0.005f)
Compare two indices via query for similarity, with a user-specified set of query vectors
-
void compareIndices(faiss::Index &refIndex, faiss::Index &testIndex, int numQuery, int dim, int k, const std::string &configMsg, float maxRelativeError = 6e-5f, float pctMaxDiff1 = 0.1f, float pctMaxDiffN = 0.005f)
Compare two indices via query for similarity, generating random query vectors
-
void compareLists(const float *refDist, const faiss::idx_t *refInd, const float *testDist, const faiss::idx_t *testInd, int dim1, int dim2, const std::string &configMsg, bool printBasicStats, bool printDiffs, bool assertOnErr, float maxRelativeError = 6e-5f, float pctMaxDiff1 = 0.1f, float pctMaxDiffN = 0.005f)
Display specific differences in the two (distance, index) lists.
-
template<typename A, typename B>
void testIVFEquality(A &cpuIndex, B &gpuIndex) Compare IVF lists between a CPU and GPU index.
-
int getCurrentDevice()
Returns the current thread-local GPU device.
-
void setCurrentDevice(int device)
Sets the current thread-local GPU device.
-
int getNumDevices()
Returns the number of available GPU devices.
-
void profilerStart()
Starts the CUDA profiler (exposed via SWIG)
-
void profilerStop()
Stops the CUDA profiler (exposed via SWIG)
-
void synchronizeAllDevices()
Synchronizes the CPU against all devices (equivalent to cudaDeviceSynchronize for each device)
-
const cudaDeviceProp &getDeviceProperties(int device)
Returns a cached cudaDeviceProp for the given device.
-
const cudaDeviceProp &getCurrentDeviceProperties()
Returns the cached cudaDeviceProp for the current device.
-
int getMaxThreads(int device)
Returns the maximum number of threads available for the given GPU device
-
int getMaxThreadsCurrentDevice()
Equivalent to getMaxThreads(getCurrentDevice())
-
dim3 getMaxGrid(int device)
Returns the maximum grid size for the given GPU device.
-
dim3 getMaxGridCurrentDevice()
Equivalent to getMaxGrid(getCurrentDevice())
Returns the maximum smem available for the given GPU device.
Equivalent to getMaxSharedMemPerBlock(getCurrentDevice())
-
int getDeviceForAddress(const void *p)
For a given pointer, returns whether or not it is located on a device (deviceId >= 0) or the host (-1).
-
bool getFullUnifiedMemSupport(int device)
Does the given device support full unified memory sharing host memory?
-
bool getFullUnifiedMemSupportCurrentDevice()
Equivalent to getFullUnifiedMemSupport(getCurrentDevice())
-
bool getTensorCoreSupport(int device)
Does the given device support tensor core operations?
-
bool getTensorCoreSupportCurrentDevice()
Equivalent to getTensorCoreSupport(getCurrentDevice())
-
size_t getFreeMemory(int device)
Returns the amount of currently available memory on the given device.
-
size_t getFreeMemoryCurrentDevice()
Equivalent to getFreeMemory(getCurrentDevice())
-
template<typename L1, typename L2>
void streamWaitBase(const L1 &listWaiting, const L2 &listWaitOn) Call for a collection of streams to wait on.
-
template<typename L1>
void streamWait(const L1 &a, const std::initializer_list<cudaStream_t> &b) These versions allow usage of initializer_list as arguments, since otherwise {…} doesn’t have a type
-
inline void streamWait(const std::initializer_list<cudaStream_t> &a, const std::initializer_list<cudaStream_t> &b)
-
inline raft::distance::DistanceType metricFaissToRaft(MetricType metric, bool exactDistance)
-
void validRowIndices(GpuResources *res, Tensor<float, 2, true> &vecs, bool *validRows)
Identify matrix rows containing non NaN values. validRows[i] is false if row i contains a NaN value and true otherwise.
-
idx_t inplaceGatherFilteredRows(GpuResources *res, Tensor<float, 2, true> &vecs, Tensor<idx_t, 1, true> &indices)
Filter out matrix rows containing NaN values. The vectors and indices are updated in-place.
-
struct GpuParameterSpace : public faiss::ParameterSpace
- #include <GpuAutoTune.h>
parameter space and setters for GPU indexes
Public Functions
-
virtual void initialize(const faiss::Index *index) override
initialize with reasonable parameters for the index
-
virtual void set_index_parameter(faiss::Index *index, const std::string &name, double val) const override
set a combination of parameters on an index
-
size_t n_combinations() const
nb of combinations, = product of values sizes
-
bool combination_ge(size_t c1, size_t c2) const
returns whether combinations c1 >= c2 in the tuple sense
-
void display() const
print a description on stdout
-
ParameterRange &add_range(const std::string &name)
add a new parameter (or return it if it exists)
-
void set_index_parameters(Index *index, size_t cno) const
set a combination of parameters on an index
-
void set_index_parameters(Index *index, const char *param_string) const
set a combination of parameters described by a string
-
void update_bounds(size_t cno, const OperatingPoint &op, double *upper_bound_perf, double *lower_bound_t) const
find an upper bound on the performance and a lower bound on t for configuration cno given another operating point op
-
void explore(Index *index, size_t nq, const float *xq, const AutoTuneCriterion &crit, OperatingPoints *ops) const
explore operating points
- Parameters:
index – index to run on
xq – query vectors (size nq * index.d)
crit – selection criterion
ops – resulting operating points
Public Members
-
std::vector<ParameterRange> parameter_ranges
all tunable parameters
-
int verbose
verbosity during exploration
-
int n_experiments
nb of experiments during optimization (0 = try all combinations)
-
size_t batchsize
maximum number of queries to submit at a time.
-
bool thread_over_batches
use multithreading over batches (useful to benchmark independent single-searches)
-
double min_test_duration
run tests several times until they reach at least this duration (to avoid jittering in MT mode)
-
virtual void initialize(const faiss::Index *index) override
-
struct ToCPUCloner : public faiss::Cloner
- #include <GpuCloner.h>
Cloner specialized for GPU -> CPU.
Public Functions
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
struct ToGpuCloner : public faiss::Cloner, public faiss::gpu::GpuClonerOptions
- #include <GpuCloner.h>
Cloner specialized for CPU -> 1 GPU.
Public Functions
-
ToGpuCloner(GpuResourcesProvider *prov, int device, const GpuClonerOptions &options)
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
Public Members
-
GpuResourcesProvider *provider
-
int device
-
IndicesOptions indicesOptions = INDICES_64_BIT
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer = false
is the coarse quantizer in float16?
-
bool useFloat16 = false
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed = false
use precomputed tables?
-
long reserveVecs = 0
reserve vectors in the invfiles?
-
bool storeTransposed = false
For GpuIndexFlat, store data in transposed layout?
-
bool verbose = false
Set verbose options on the index.
-
bool use_raft = false
use the RAFT implementation
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
ToGpuCloner(GpuResourcesProvider *prov, int device, const GpuClonerOptions &options)
-
struct ToGpuClonerMultiple : public faiss::Cloner, public faiss::gpu::GpuMultipleClonerOptions
- #include <GpuCloner.h>
Cloner specialized for CPU -> multiple GPUs.
Public Functions
-
ToGpuClonerMultiple(std::vector<GpuResourcesProvider*> &provider, std::vector<int> &devices, const GpuMultipleClonerOptions &options)
-
ToGpuClonerMultiple(const std::vector<ToGpuCloner> &sub_cloners, const GpuMultipleClonerOptions &options)
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
Public Members
-
std::vector<ToGpuCloner> sub_cloners
-
bool shard = false
Whether to shard the index across GPUs, versus replication across GPUs
-
int shard_type = 1
IndexIVF::copy_subset_to subset type.
-
bool common_ivf_quantizer = false
set to true if an IndexIVF is to be dispatched to multiple GPUs with a single common IVF quantizer, ie. only the inverted lists are sharded on the sub-indexes (uses an IndexShardsIVF)
-
IndicesOptions indicesOptions = INDICES_64_BIT
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer = false
is the coarse quantizer in float16?
-
bool useFloat16 = false
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed = false
use precomputed tables?
-
long reserveVecs = 0
reserve vectors in the invfiles?
-
bool storeTransposed = false
For GpuIndexFlat, store data in transposed layout?
-
bool verbose = false
Set verbose options on the index.
-
bool use_raft = false
use the RAFT implementation
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
ToGpuClonerMultiple(std::vector<GpuResourcesProvider*> &provider, std::vector<int> &devices, const GpuMultipleClonerOptions &options)
-
struct GpuProgressiveDimIndexFactory : public faiss::ProgressiveDimIndexFactory
- #include <GpuCloner.h>
index factory for the ProgressiveDimClustering object
Public Functions
-
explicit GpuProgressiveDimIndexFactory(int ngpu)
-
virtual Index *operator()(int dim) override
ownership transferred to caller
-
virtual ~GpuProgressiveDimIndexFactory() override
Public Members
-
GpuMultipleClonerOptions options
-
std::vector<GpuResourcesProvider*> vres
-
std::vector<int> devices
-
int ncall
-
explicit GpuProgressiveDimIndexFactory(int ngpu)
-
struct GpuClonerOptions
- #include <GpuClonerOptions.h>
set some options on how to copy to GPU
Subclassed by faiss::gpu::GpuMultipleClonerOptions, faiss::gpu::ToGpuCloner
Public Members
-
IndicesOptions indicesOptions = INDICES_64_BIT
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer = false
is the coarse quantizer in float16?
-
bool useFloat16 = false
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed = false
use precomputed tables?
-
long reserveVecs = 0
reserve vectors in the invfiles?
-
bool storeTransposed = false
For GpuIndexFlat, store data in transposed layout?
-
bool verbose = false
Set verbose options on the index.
-
bool use_raft = false
use the RAFT implementation
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
IndicesOptions indicesOptions = INDICES_64_BIT
-
struct GpuMultipleClonerOptions : public faiss::gpu::GpuClonerOptions
Subclassed by faiss::gpu::ToGpuClonerMultiple
Public Members
-
bool shard = false
Whether to shard the index across GPUs, versus replication across GPUs
-
int shard_type = 1
IndexIVF::copy_subset_to subset type.
-
bool common_ivf_quantizer = false
set to true if an IndexIVF is to be dispatched to multiple GPUs with a single common IVF quantizer, ie. only the inverted lists are sharded on the sub-indexes (uses an IndexShardsIVF)
-
IndicesOptions indicesOptions = INDICES_64_BIT
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer = false
is the coarse quantizer in float16?
-
bool useFloat16 = false
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed = false
use precomputed tables?
-
long reserveVecs = 0
reserve vectors in the invfiles?
-
bool storeTransposed = false
For GpuIndexFlat, store data in transposed layout?
-
bool verbose = false
Set verbose options on the index.
-
bool use_raft = false
use the RAFT implementation
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
bool shard = false
-
struct GpuDistanceParams
- #include <GpuDistance.h>
Arguments to brute-force GPU k-nearest neighbor searching.
Public Members
-
faiss::MetricType metric = METRIC_L2
Search parameter: distance metric.
-
float metricArg = 0
Search parameter: distance metric argument (if applicable) For metric == METRIC_Lp, this is the p-value
-
int k = 0
Search parameter: return k nearest neighbors If the value provided is -1, then we report all pairwise distances without top-k filtering
-
int dims = 0
Vector dimensionality.
-
const void *vectors = nullptr
If vectorsRowMajor is true, this is numVectors x dims, with dims innermost; otherwise, dims x numVectors, with numVectors innermost
-
DistanceDataType vectorType = DistanceDataType::F32
-
bool vectorsRowMajor = true
-
idx_t numVectors = 0
-
const float *vectorNorms = nullptr
Precomputed L2 norms for each vector in
vectors, which can be optionally provided in advance to speed computation for METRIC_L2
-
const void *queries = nullptr
If queriesRowMajor is true, this is numQueries x dims, with dims innermost; otherwise, dims x numQueries, with numQueries innermost
-
DistanceDataType queryType = DistanceDataType::F32
-
bool queriesRowMajor = true
-
idx_t numQueries = 0
-
float *outDistances = nullptr
A region of memory size numQueries x k, with k innermost (row major) if k > 0, or if k == -1, a region of memory of size numQueries x numVectors
-
bool ignoreOutDistances = false
Do we only care about the indices reported, rather than the output distances? Not used if k == -1 (all pairwise distances)
-
IndicesDataType outIndicesType = IndicesDataType::I64
A region of memory size numQueries x k, with k innermost (row major). Not used if k == -1 (all pairwise distances)
-
void *outIndices = nullptr
-
int device = -1
On which GPU device should the search run? -1 indicates that the current CUDA thread-local device (via cudaGetDevice/cudaSetDevice) is used Otherwise, an integer 0 <= device < numDevices indicates the device for execution
-
bool use_raft = false
Should the index dispatch down to RAFT? TODO: change default to true if RAFT is enabled
-
faiss::MetricType metric = METRIC_L2
-
class GpuIcmEncoder : public faiss::lsq::IcmEncoder
- #include <GpuIcmEncoder.h>
Perform LSQ encoding on GPU.
Split input vectors to different devices and call IcmEncoderImpl::encode to encode them
Public Functions
-
GpuIcmEncoder(const LocalSearchQuantizer *lsq, const std::vector<GpuResourcesProvider*> &provs, const std::vector<int> &devices)
-
~GpuIcmEncoder()
-
GpuIcmEncoder(const GpuIcmEncoder&) = delete
-
GpuIcmEncoder &operator=(const GpuIcmEncoder&) = delete
-
virtual void set_binary_term() override
-
virtual void encode(int32_t *codes, const float *x, std::mt19937 &gen, size_t n, size_t ils_iters) const override
Encode vectors given codebooks
- Parameters:
codes – output codes, size n * M
x – vectors to encode, size n * d
gen – random generator
n – number of vectors
ils_iters – number of iterations of iterative local search
Private Members
-
std::unique_ptr<IcmEncoderShards> shards
-
GpuIcmEncoder(const LocalSearchQuantizer *lsq, const std::vector<GpuResourcesProvider*> &provs, const std::vector<int> &devices)
-
struct GpuIcmEncoderFactory : public faiss::lsq::IcmEncoderFactory
Public Functions
-
explicit GpuIcmEncoderFactory(int ngpus = 1)
-
virtual lsq::IcmEncoder *get(const LocalSearchQuantizer *lsq) override
-
explicit GpuIcmEncoderFactory(int ngpus = 1)
-
struct GpuIndexConfig
Subclassed by faiss::gpu::GpuIndexBinaryFlatConfig, faiss::gpu::GpuIndexFlatConfig, faiss::gpu::GpuIndexIVFConfig
Public Members
-
int device = 0
GPU device on which the index is resident.
-
MemorySpace memorySpace = MemorySpace::Device
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
bool use_raft = false
Should the index dispatch down to RAFT?
-
int device = 0
-
class GpuIndex : public faiss::Index
Subclassed by faiss::gpu::GpuIndexFlat, faiss::gpu::GpuIndexIVF
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reset() = 0
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual bool addImplRequiresIDs_() const = 0
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) = 0
Overridden to actually perform the add All data is guaranteed to be resident on our device
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const = 0
Overridden to actually perform the search All data is guaranteed to be resident on our device
Protected Attributes
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
Private Functions
-
void addPaged_(idx_t n, const float *x, const idx_t *ids)
Handles paged adds if the add set is too large, passes to addImpl_ to actually perform the add for the current page
-
void addPage_(idx_t n, const float *x, const idx_t *ids)
Calls addImpl_ for a single page of GPU-resident data.
-
void searchNonPaged_(idx_t n, const float *x, int k, float *outDistancesData, idx_t *outIndicesData, const SearchParameters *params) const
Calls searchImpl_ for a single page of GPU-resident data.
-
void searchFromCpuPaged_(idx_t n, const float *x, int k, float *outDistancesData, idx_t *outIndicesData, const SearchParameters *params) const
Calls searchImpl_ for a single page of GPU-resident data, handling paging of the data and copies from the CPU
-
using component_t = float
-
struct GpuIndexBinaryFlatConfig : public faiss::gpu::GpuIndexConfig
Public Members
-
int device = 0
GPU device on which the index is resident.
-
MemorySpace memorySpace = MemorySpace::Device
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
bool use_raft = false
Should the index dispatch down to RAFT?
-
int device = 0
-
class GpuIndexBinaryFlat : public faiss::IndexBinary
- #include <GpuIndexBinaryFlat.h>
A GPU version of IndexBinaryFlat for brute-force comparison of bit vectors via Hamming distance
Public Types
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
GpuIndexBinaryFlat(GpuResourcesProvider *resources, const faiss::IndexBinaryFlat *index, GpuIndexBinaryFlatConfig config = GpuIndexBinaryFlatConfig())
Construct from a pre-existing faiss::IndexBinaryFlat instance, copying data over to the given GPU
-
GpuIndexBinaryFlat(GpuResourcesProvider *resources, int dims, GpuIndexBinaryFlatConfig config = GpuIndexBinaryFlatConfig())
Construct an empty instance that can be added to.
-
~GpuIndexBinaryFlat() override
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void copyFrom(const faiss::IndexBinaryFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexBinaryFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
virtual void add(faiss::idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, faiss::idx_t *labels, const faiss::SearchParameters *params = nullptr) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void reconstruct(faiss::idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons, const SearchParameters *params = nullptr) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
-
virtual void merge_from(IndexBinary &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const IndexBinary &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
int d = 0
vector dimension
-
int code_size = 0
number of bytes per vector ( = d / 8 )
-
idx_t ntotal = 0
total nb of indexed vectors
-
bool verbose = false
verbosity level
-
bool is_trained = true
set if the Index does not require training, or if training is done already
-
MetricType metric_type = METRIC_L2
type of metric this index uses for search
Protected Functions
Protected Attributes
-
std::shared_ptr<GpuResources> resources_
Manages streans, cuBLAS handles and scratch memory for devices.
-
const GpuIndexBinaryFlatConfig binaryFlatConfig_
Configuration options.
-
std::unique_ptr<BinaryFlatIndex> data_
Holds our GPU data containing the list of vectors.
-
using component_t = uint8_t
-
struct GpuIndexFlatConfig : public faiss::gpu::GpuIndexConfig
Public Functions
- bool ALIGNED (8) useFloat16
Whether or not data is stored as float16.
Public Members
-
bool storeTransposed = false
Deprecated: no longer used Previously used to indicate whether internal storage of vectors is transposed
-
int device = 0
GPU device on which the index is resident.
-
MemorySpace memorySpace = MemorySpace::Device
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
bool use_raft = false
Should the index dispatch down to RAFT?
-
class GpuIndexFlat : public faiss::gpu::GpuIndex
- #include <GpuIndexFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexFlat; copies over centroid data from a given faiss::IndexFlat
Subclassed by faiss::gpu::GpuIndexFlatIP, faiss::gpu::GpuIndexFlatL2
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexFlat(GpuResourcesProvider *provider, const faiss::IndexFlat *index, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct from a pre-existing faiss::IndexFlat instance, copying data over to the given GPU
-
GpuIndexFlat(GpuResourcesProvider *provider, int dims, faiss::MetricType metric, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct an empty instance that can be added to.
-
~GpuIndexFlat() override
-
void copyFrom(const faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t getNumVecs() const
Returns the number of vectors we contain.
-
virtual void reset() override
Clears all vectors from this index.
-
virtual void train(idx_t n, const float *x) override
This index is not trained, so this does nothing.
-
virtual void add(idx_t, const float *x) override
Overrides to avoid excessive copies.
-
virtual void reconstruct(idx_t key, float *out) const override
Reconstruction methods; prefer the batch reconstruct as it will be more efficient
-
virtual void reconstruct_n(idx_t i0, idx_t num, float *out) const override
Batch reconstruction method.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *out) const override
Batch reconstruction method.
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Compute residual.
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Compute residual (batch mode)
-
inline FlatIndex *getGpuData()
For internal access.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
void resetIndex_(int dims)
-
virtual bool addImplRequiresIDs_() const override
Flat index does not require IDs as there is no storage available for them
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) override
Called from GpuIndex for add.
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const override
Called from GpuIndex for search.
Protected Attributes
-
const GpuIndexFlatConfig flatConfig_
Our configuration options.
-
std::unique_ptr<FlatIndex> data_
Holds our GPU data containing the list of vectors.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using component_t = float
-
class GpuIndexFlatL2 : public faiss::gpu::GpuIndexFlat
- #include <GpuIndexFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexFlatL2; copies over centroid data from a given faiss::IndexFlat
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexFlatL2(GpuResourcesProvider *provider, faiss::IndexFlatL2 *index, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct from a pre-existing faiss::IndexFlatL2 instance, copying data over to the given GPU
-
GpuIndexFlatL2(GpuResourcesProvider *provider, int dims, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct an empty instance that can be added to.
-
void copyFrom(faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index)
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
void copyFrom(const faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t getNumVecs() const
Returns the number of vectors we contain.
-
virtual void reset() override
Clears all vectors from this index.
-
virtual void train(idx_t n, const float *x) override
This index is not trained, so this does nothing.
-
virtual void add(idx_t, const float *x) override
Overrides to avoid excessive copies.
-
virtual void reconstruct(idx_t key, float *out) const override
Reconstruction methods; prefer the batch reconstruct as it will be more efficient
-
virtual void reconstruct_n(idx_t i0, idx_t num, float *out) const override
Batch reconstruction method.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *out) const override
Batch reconstruction method.
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Compute residual.
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Compute residual (batch mode)
-
inline FlatIndex *getGpuData()
For internal access.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
void resetIndex_(int dims)
-
virtual bool addImplRequiresIDs_() const override
Flat index does not require IDs as there is no storage available for them
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) override
Called from GpuIndex for add.
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const override
Called from GpuIndex for search.
Protected Attributes
-
const GpuIndexFlatConfig flatConfig_
Our configuration options.
-
std::unique_ptr<FlatIndex> data_
Holds our GPU data containing the list of vectors.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using component_t = float
-
class GpuIndexFlatIP : public faiss::gpu::GpuIndexFlat
- #include <GpuIndexFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexFlatIP; copies over centroid data from a given faiss::IndexFlat
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexFlatIP(GpuResourcesProvider *provider, faiss::IndexFlatIP *index, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct from a pre-existing faiss::IndexFlatIP instance, copying data over to the given GPU
-
GpuIndexFlatIP(GpuResourcesProvider *provider, int dims, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct an empty instance that can be added to.
-
void copyFrom(faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index)
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
void copyFrom(const faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t getNumVecs() const
Returns the number of vectors we contain.
-
virtual void reset() override
Clears all vectors from this index.
-
virtual void train(idx_t n, const float *x) override
This index is not trained, so this does nothing.
-
virtual void add(idx_t, const float *x) override
Overrides to avoid excessive copies.
-
virtual void reconstruct(idx_t key, float *out) const override
Reconstruction methods; prefer the batch reconstruct as it will be more efficient
-
virtual void reconstruct_n(idx_t i0, idx_t num, float *out) const override
Batch reconstruction method.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *out) const override
Batch reconstruction method.
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Compute residual.
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Compute residual (batch mode)
-
inline FlatIndex *getGpuData()
For internal access.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
void resetIndex_(int dims)
-
virtual bool addImplRequiresIDs_() const override
Flat index does not require IDs as there is no storage available for them
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) override
Called from GpuIndex for add.
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const override
Called from GpuIndex for search.
Protected Attributes
-
const GpuIndexFlatConfig flatConfig_
Our configuration options.
-
std::unique_ptr<FlatIndex> data_
Holds our GPU data containing the list of vectors.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using component_t = float
-
struct GpuIndexIVFConfig : public faiss::gpu::GpuIndexConfig
Subclassed by faiss::gpu::GpuIndexIVFFlatConfig, faiss::gpu::GpuIndexIVFPQConfig, faiss::gpu::GpuIndexIVFScalarQuantizerConfig
Public Members
-
IndicesOptions indicesOptions = INDICES_64_BIT
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig
Configuration for the coarse quantizer object.
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
int device = 0
GPU device on which the index is resident.
-
MemorySpace memorySpace = MemorySpace::Device
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
bool use_raft = false
Should the index dispatch down to RAFT?
-
IndicesOptions indicesOptions = INDICES_64_BIT
-
class GpuIndexIVF : public faiss::gpu::GpuIndex, public faiss::IndexIVFInterface
- #include <GpuIndexIVF.h>
Base class of all GPU IVF index types. This (for now) deliberately does not inherit from IndexIVF, as many of the public data members and functionality in IndexIVF is not supported in the same manner on the GPU.
Subclassed by faiss::gpu::GpuIndexIVFFlat, faiss::gpu::GpuIndexIVFPQ, faiss::gpu::GpuIndexIVFScalarQuantizer
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVF(GpuResourcesProvider *provider, int dims, faiss::MetricType metric, float metricArg, idx_t nlist, GpuIndexIVFConfig config = GpuIndexIVFConfig())
Version that auto-constructs a flat coarse quantizer based on the desired metric
-
GpuIndexIVF(GpuResourcesProvider *provider, Index *coarseQuantizer, int dims, faiss::MetricType metric, float metricArg, idx_t nlist, GpuIndexIVFConfig config = GpuIndexIVFConfig())
Version that takes a coarse quantizer instance. The GpuIndexIVF does not own the coarseQuantizer instance by default (functions like IndexIVF).
-
~GpuIndexIVF() override
-
virtual void updateQuantizer() = 0
Should be called if the user ever changes the state of the IVF coarse quantizer manually (e.g., substitutes a new instance or changes vectors in the coarse quantizer outside the scope of training)
-
virtual idx_t getNumLists() const
Returns the number of inverted lists we’re managing.
-
virtual idx_t getListLength(idx_t listId) const
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(idx_t listId, bool gpuFormat = false) const
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<idx_t> getListIndices(idx_t listId) const
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const SearchParametersIVF *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reset() = 0
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
Protected Functions
-
int getCurrentNProbe_(const SearchParameters *params) const
From either the current set nprobe or the SearchParameters if available, return the nprobe that we should use for the current search
-
void verifyIVFSettings_() const
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
-
virtual void trainQuantizer_(idx_t n, const float *x)
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) override
Called from GpuIndex for add/add_with_ids.
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const override
Called from GpuIndex for search.
Protected Attributes
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<IVFBase> baseIndex_
For a trained/initialized index, this is a reference to the base class.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
Private Functions
-
void init_()
Shared initialization functions.
-
using component_t = float
-
struct GpuIndexIVFFlatConfig : public faiss::gpu::GpuIndexIVFConfig
Public Members
-
bool interleavedLayout = true
Use the alternative memory layout for the IVF lists (currently the default)
-
IndicesOptions indicesOptions = INDICES_64_BIT
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig
Configuration for the coarse quantizer object.
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
int device = 0
GPU device on which the index is resident.
-
MemorySpace memorySpace = MemorySpace::Device
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
bool use_raft = false
Should the index dispatch down to RAFT?
-
bool interleavedLayout = true
-
class GpuIndexIVFFlat : public faiss::gpu::GpuIndexIVF
- #include <GpuIndexIVFFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexIVFFlat
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVFFlat(GpuResourcesProvider *provider, const faiss::IndexIVFFlat *index, GpuIndexIVFFlatConfig config = GpuIndexIVFFlatConfig())
Construct from a pre-existing faiss::IndexIVFFlat instance, copying data over to the given GPU, if the input index is trained.
-
GpuIndexIVFFlat(GpuResourcesProvider *provider, int dims, idx_t nlist, faiss::MetricType metric = faiss::METRIC_L2, GpuIndexIVFFlatConfig config = GpuIndexIVFFlatConfig())
Constructs a new instance with an empty flat quantizer; the user provides the number of IVF lists desired.
-
GpuIndexIVFFlat(GpuResourcesProvider *provider, Index *coarseQuantizer, int dims, idx_t nlist, faiss::MetricType metric = faiss::METRIC_L2, GpuIndexIVFFlatConfig config = GpuIndexIVFFlatConfig())
Constructs a new instance with a provided CPU or GPU coarse quantizer; the user provides the number of IVF lists desired.
-
~GpuIndexIVFFlat() override
-
void reserveMemory(size_t numVecs)
Reserve GPU memory in our inverted lists for this number of vectors.
-
void copyFrom(const faiss::IndexIVFFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexIVFFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t reclaimMemory()
After adding vectors, one can call this to reclaim device memory to exactly the amount needed. Returns space reclaimed in bytes
-
virtual void reset() override
Clears out all inverted lists, but retains the coarse centroid information
-
virtual void updateQuantizer() override
Should be called if the user ever changes the state of the IVF coarse quantizer manually (e.g., substitutes a new instance or changes vectors in the coarse quantizer outside the scope of training)
-
virtual void train(idx_t n, const float *x) override
Trains the coarse quantizer based on the given vector data.
-
virtual void reconstruct_n(idx_t i0, idx_t n, float *out) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual idx_t getNumLists() const
Returns the number of inverted lists we’re managing.
-
virtual idx_t getListLength(idx_t listId) const
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(idx_t listId, bool gpuFormat = false) const
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<idx_t> getListIndices(idx_t listId) const
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const SearchParametersIVF *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
Protected Functions
-
void setIndex_(GpuResources *resources, int dim, int nlist, faiss::MetricType metric, float metricArg, bool useResidual, faiss::ScalarQuantizer *scalarQ, bool interleavedLayout, IndicesOptions indicesOptions, MemorySpace space)
Initialize appropriate index.
- Parameters:
scalarQ – Optional ScalarQuantizer
-
int getCurrentNProbe_(const SearchParameters *params) const
From either the current set nprobe or the SearchParameters if available, return the nprobe that we should use for the current search
-
void verifyIVFSettings_() const
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
-
virtual void trainQuantizer_(idx_t n, const float *x)
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) override
Called from GpuIndex for add/add_with_ids.
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const override
Called from GpuIndex for search.
Protected Attributes
-
const GpuIndexIVFFlatConfig ivfFlatConfig_
Our configuration options.
-
size_t reserveMemoryVecs_
Desired inverted list memory reservation.
-
std::shared_ptr<IVFFlat> index_
Instance that we own; contains the inverted lists.
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<IVFBase> baseIndex_
For a trained/initialized index, this is a reference to the base class.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using component_t = float
-
struct GpuIndexIVFPQConfig : public faiss::gpu::GpuIndexIVFConfig
Public Members
-
bool useFloat16LookupTables = false
Whether or not float16 residual distance tables are used in the list scanning kernels. When subQuantizers * 2^bitsPerCode > 16384, this is required.
-
bool usePrecomputedTables = false
Whether or not we enable the precomputed table option for search, which can substantially increase the memory requirement.
-
bool interleavedLayout = false
Use the alternative memory layout for the IVF lists WARNING: this is a feature under development, and is only supported with RAFT enabled for the index. Do not use if RAFT is not enabled.
-
bool useMMCodeDistance = false
Use GEMM-backed computation of PQ code distances for the no precomputed table version of IVFPQ. This is for debugging purposes, it should not substantially affect the results one way for another.
Note that MM code distance is enabled automatically if one uses a number of dimensions per sub-quantizer that is not natively specialized (an odd number like 7 or so).
-
IndicesOptions indicesOptions = INDICES_64_BIT
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig
Configuration for the coarse quantizer object.
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
int device = 0
GPU device on which the index is resident.
-
MemorySpace memorySpace = MemorySpace::Device
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
bool use_raft = false
Should the index dispatch down to RAFT?
-
bool useFloat16LookupTables = false
-
class GpuIndexIVFPQ : public faiss::gpu::GpuIndexIVF
- #include <GpuIndexIVFPQ.h>
IVFPQ index for the GPU.
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVFPQ(GpuResourcesProvider *provider, const faiss::IndexIVFPQ *index, GpuIndexIVFPQConfig config = GpuIndexIVFPQConfig())
Construct from a pre-existing faiss::IndexIVFPQ instance, copying data over to the given GPU, if the input index is trained.
-
GpuIndexIVFPQ(GpuResourcesProvider *provider, int dims, idx_t nlist, idx_t subQuantizers, idx_t bitsPerCode, faiss::MetricType metric = faiss::METRIC_L2, GpuIndexIVFPQConfig config = GpuIndexIVFPQConfig())
Constructs a new instance with an empty flat quantizer; the user provides the number of IVF lists desired.
-
GpuIndexIVFPQ(GpuResourcesProvider *provider, Index *coarseQuantizer, int dims, idx_t nlist, idx_t subQuantizers, idx_t bitsPerCode, faiss::MetricType metric = faiss::METRIC_L2, GpuIndexIVFPQConfig config = GpuIndexIVFPQConfig())
Constructs a new instance with a provided CPU or GPU coarse quantizer; the user provides the number of IVF lists desired.
-
~GpuIndexIVFPQ() override
-
void copyFrom(const faiss::IndexIVFPQ *index)
Reserve space on the GPU for the inverted lists for
numvectors, assumed equally distributed among Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexIVFPQ *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
void reserveMemory(size_t numVecs)
Reserve GPU memory in our inverted lists for this number of vectors.
-
void setPrecomputedCodes(bool enable)
Enable or disable pre-computed codes.
-
bool getPrecomputedCodes() const
Are pre-computed codes enabled?
-
int getNumSubQuantizers() const
Return the number of sub-quantizers we are using.
-
int getBitsPerCode() const
Return the number of bits per PQ code.
-
int getCentroidsPerSubQuantizer() const
Return the number of centroids per PQ code (2^bits per code)
-
size_t reclaimMemory()
After adding vectors, one can call this to reclaim device memory to exactly the amount needed. Returns space reclaimed in bytes
-
virtual void reset() override
Clears out all inverted lists, but retains the coarse and product centroid information
-
virtual void updateQuantizer() override
Should be called if the user ever changes the state of the IVF coarse quantizer manually (e.g., substitutes a new instance or changes vectors in the coarse quantizer outside the scope of training)
-
virtual void train(idx_t n, const float *x) override
Trains the coarse and product quantizer based on the given vector data.
-
virtual idx_t getNumLists() const
Returns the number of inverted lists we’re managing.
-
virtual idx_t getListLength(idx_t listId) const
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(idx_t listId, bool gpuFormat = false) const
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<idx_t> getListIndices(idx_t listId) const
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const SearchParametersIVF *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
ProductQuantizer pq
Like the CPU version, we expose a publically-visible ProductQuantizer for manipulation
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
Protected Functions
-
void setIndex_(GpuResources *resources, int dim, idx_t nlist, faiss::MetricType metric, float metricArg, int numSubQuantizers, int bitsPerSubQuantizer, bool useFloat16LookupTables, bool useMMCodeDistance, bool interleavedLayout, float *pqCentroidData, IndicesOptions indicesOptions, MemorySpace space)
Initialize appropriate index.
-
void verifyPQSettings_() const
Throws errors if configuration settings are improper.
-
void trainResidualQuantizer_(idx_t n, const float *x)
Trains the PQ quantizer based on the given vector data.
-
int getCurrentNProbe_(const SearchParameters *params) const
From either the current set nprobe or the SearchParameters if available, return the nprobe that we should use for the current search
-
void verifyIVFSettings_() const
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
-
virtual void trainQuantizer_(idx_t n, const float *x)
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) override
Called from GpuIndex for add/add_with_ids.
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const override
Called from GpuIndex for search.
Protected Attributes
-
const GpuIndexIVFPQConfig ivfpqConfig_
Our configuration options that we were initialized with.
-
bool usePrecomputedTables_
Runtime override: whether or not we use precomputed tables.
-
int subQuantizers_
Number of sub-quantizers per encoded vector.
-
int bitsPerCode_
Bits per sub-quantizer code.
-
size_t reserveMemoryVecs_
Desired inverted list memory reservation.
-
std::shared_ptr<IVFPQ> index_
The product quantizer instance that we own; contains the inverted lists
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<IVFBase> baseIndex_
For a trained/initialized index, this is a reference to the base class.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using component_t = float
-
struct GpuIndexIVFScalarQuantizerConfig : public faiss::gpu::GpuIndexIVFConfig
Public Members
-
bool interleavedLayout = true
Use the alternative memory layout for the IVF lists (currently the default)
-
IndicesOptions indicesOptions = INDICES_64_BIT
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig
Configuration for the coarse quantizer object.
-
bool allowCpuCoarseQuantizer = false
This flag controls the CPU fallback logic for coarse quantizer component of the index. When set to false (default), the cloner will throw an exception for indices not implemented on GPU. When set to true, it will fallback to a CPU implementation.
-
int device = 0
GPU device on which the index is resident.
-
MemorySpace memorySpace = MemorySpace::Device
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
bool use_raft = false
Should the index dispatch down to RAFT?
-
bool interleavedLayout = true
-
class GpuIndexIVFScalarQuantizer : public faiss::gpu::GpuIndexIVF
- #include <GpuIndexIVFScalarQuantizer.h>
Wrapper around the GPU implementation that looks like faiss::IndexIVFScalarQuantizer
Public Types
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVFScalarQuantizer(GpuResourcesProvider *provider, const faiss::IndexIVFScalarQuantizer *index, GpuIndexIVFScalarQuantizerConfig config = GpuIndexIVFScalarQuantizerConfig())
Construct from a pre-existing faiss::IndexIVFScalarQuantizer instance, copying data over to the given GPU, if the input index is trained.
-
GpuIndexIVFScalarQuantizer(GpuResourcesProvider *provider, int dims, idx_t nlist, faiss::ScalarQuantizer::QuantizerType qtype, faiss::MetricType metric = MetricType::METRIC_L2, bool encodeResidual = true, GpuIndexIVFScalarQuantizerConfig config = GpuIndexIVFScalarQuantizerConfig())
Constructs a new instance with an empty flat quantizer; the user provides the number of IVF lists desired.
-
GpuIndexIVFScalarQuantizer(GpuResourcesProvider *provider, Index *coarseQuantizer, int dims, idx_t nlist, faiss::ScalarQuantizer::QuantizerType qtype, faiss::MetricType metric = MetricType::METRIC_L2, bool encodeResidual = true, GpuIndexIVFScalarQuantizerConfig config = GpuIndexIVFScalarQuantizerConfig())
Constructs a new instance with a provided CPU or GPU coarse quantizer; the user provides the number of IVF lists desired.
-
~GpuIndexIVFScalarQuantizer() override
-
void reserveMemory(size_t numVecs)
Reserve GPU memory in our inverted lists for this number of vectors.
-
void copyFrom(const faiss::IndexIVFScalarQuantizer *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexIVFScalarQuantizer *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t reclaimMemory()
After adding vectors, one can call this to reclaim device memory to exactly the amount needed. Returns space reclaimed in bytes
-
virtual void reset() override
Clears out all inverted lists, but retains the coarse and scalar quantizer information
-
virtual void updateQuantizer() override
Should be called if the user ever changes the state of the IVF coarse quantizer manually (e.g., substitutes a new instance or changes vectors in the coarse quantizer outside the scope of training)
-
virtual void train(idx_t n, const float *x) override
Trains the coarse and scalar quantizer based on the given vector data.
-
virtual idx_t getNumLists() const
Returns the number of inverted lists we’re managing.
-
virtual idx_t getListLength(idx_t listId) const
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(idx_t listId, bool gpuFormat = false) const
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<idx_t> getListIndices(idx_t listId) const
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const SearchParametersIVF *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
Range search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the RangeSearchResults results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
result – Output results
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons, const SearchParameters *params = nullptr) const override
x,distancesandlabelsandreconscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result, const SearchParameters *params = nullptr) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
n – number of vectors
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
i0 – index of the first vector in the sequence
ni – number of vectors in the sequence
recons – reconstucted vector (size ni * d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void merge_from(Index &otherIndex, idx_t add_id = 0)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal)
-
virtual void check_compatible_for_merge(const Index &otherIndex) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
void encode_listno(idx_t list_no, uint8_t *code) const
-
idx_t decode_listno(const uint8_t *code) const
Public Members
-
faiss::ScalarQuantizer sq
Exposed like the CPU version.
-
bool by_residual
Exposed like the CPU version.
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
size_t nprobe = 1
number of probes at query time
-
size_t max_codes = 0
max nb of codes to visit to do a query
-
Index *quantizer = nullptr
quantizer that maps vectors to inverted lists
-
size_t nlist = 0
number of inverted lists
-
char quantizer_trains_alone = 0
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields = false
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index = nullptr
to override index used during clustering
Protected Functions
-
void verifySQSettings_() const
Validates index SQ parameters.
-
void trainResiduals_(idx_t n, const float *x)
Called from train to handle SQ residual training.
-
int getCurrentNProbe_(const SearchParameters *params) const
From either the current set nprobe or the SearchParameters if available, return the nprobe that we should use for the current search
-
void verifyIVFSettings_() const
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
-
virtual void trainQuantizer_(idx_t n, const float *x)
-
virtual void addImpl_(idx_t n, const float *x, const idx_t *ids) override
Called from GpuIndex for add/add_with_ids.
-
virtual void searchImpl_(idx_t n, const float *x, int k, float *distances, idx_t *labels, const SearchParameters *params) const override
Called from GpuIndex for search.
Protected Attributes
-
const GpuIndexIVFScalarQuantizerConfig ivfSQConfig_
Our configuration options.
-
size_t reserveMemoryVecs_
Desired inverted list memory reservation.
-
std::shared_ptr<IVFFlat> index_
Instance that we own; contains the inverted list.
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<IVFBase> baseIndex_
For a trained/initialized index, this is a reference to the base class.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using component_t = float
-
struct AllocInfo
- #include <GpuResources.h>
Information on what/where an allocation is.
Subclassed by faiss::gpu::AllocRequest
Public Functions
-
inline AllocInfo()
-
inline AllocInfo(AllocType at, int dev, MemorySpace sp, cudaStream_t st)
-
std::string toString() const
Returns a string representation of this info.
Public Members
-
int device = 0
The device on which the allocation is happening.
-
MemorySpace space = MemorySpace::Device
The memory space of the allocation.
-
cudaStream_t stream = nullptr
The stream on which new work on the memory will be ordered (e.g., if a piece of memory cached and to be returned for this call was last used on stream 3 and a new memory request is for stream 4, the memory manager will synchronize stream 4 to wait for the completion of stream 3 via events or other stream synchronization.
The memory manager guarantees that the returned memory is free to use without data races on this stream specified.
-
inline AllocInfo()
-
struct AllocRequest : public faiss::gpu::AllocInfo
- #include <GpuResources.h>
Information on what/where an allocation is, along with how big it should be.
Public Functions
-
inline AllocRequest()
-
inline AllocRequest(const AllocInfo &info, size_t sz)
-
inline AllocRequest(AllocType at, int dev, MemorySpace sp, cudaStream_t st, size_t sz)
-
std::string toString() const
Returns a string representation of this request.
Public Members
-
size_t size = 0
The size in bytes of the allocation.
-
int device = 0
The device on which the allocation is happening.
-
MemorySpace space = MemorySpace::Device
The memory space of the allocation.
-
cudaStream_t stream = nullptr
The stream on which new work on the memory will be ordered (e.g., if a piece of memory cached and to be returned for this call was last used on stream 3 and a new memory request is for stream 4, the memory manager will synchronize stream 4 to wait for the completion of stream 3 via events or other stream synchronization.
The memory manager guarantees that the returned memory is free to use without data races on this stream specified.
-
inline AllocRequest()
-
struct GpuMemoryReservation
- #include <GpuResources.h>
A RAII object that manages a temporary memory request.
Public Functions
-
GpuMemoryReservation()
-
GpuMemoryReservation(GpuResources *r, int dev, cudaStream_t str, void *p, size_t sz)
-
GpuMemoryReservation(GpuMemoryReservation &&m) noexcept
-
~GpuMemoryReservation()
-
GpuMemoryReservation &operator=(GpuMemoryReservation &&m)
-
inline void *get()
-
void release()
-
GpuMemoryReservation()
-
class GpuResources
- #include <GpuResources.h>
Base class of GPU-side resource provider; hides provision of cuBLAS handles, CUDA streams and all device memory allocation performed
Subclassed by faiss::gpu::StandardGpuResourcesImpl
Public Functions
-
virtual ~GpuResources()
-
virtual void initializeForDevice(int device) = 0
Call to pre-allocate resources for a particular device. If this is not called, then resources will be allocated at the first time of demand
-
virtual cublasHandle_t getBlasHandle(int device) = 0
Returns the cuBLAS handle that we use for the given device.
-
virtual cudaStream_t getDefaultStream(int device) = 0
Returns the stream that we order all computation on for the given device
-
virtual void setDefaultStream(int device, cudaStream_t stream) = 0
Overrides the default stream for a device to the user-supplied stream. The resources object does not own this stream (i.e., it will not destroy it).
-
virtual std::vector<cudaStream_t> getAlternateStreams(int device) = 0
Returns the set of alternative streams that we use for the given device.
-
virtual void *allocMemory(const AllocRequest &req) = 0
Memory management Returns an allocation from the given memory space, ordered with respect to the given stream (i.e., the first user will be a kernel in this stream). All allocations are sized internally to be the next highest multiple of 16 bytes, and all allocations returned are guaranteed to be 16 byte aligned.
-
virtual void deallocMemory(int device, void *in) = 0
Returns a previous allocation.
-
virtual size_t getTempMemoryAvailable(int device) const = 0
For MemorySpace::Temporary, how much space is immediately available without cudaMalloc allocation?
-
virtual std::pair<void*, size_t> getPinnedMemory() = 0
Returns the available CPU pinned memory buffer.
-
virtual cudaStream_t getAsyncCopyStream(int device) = 0
Returns the stream on which we perform async CPU <-> GPU copies.
-
cublasHandle_t getBlasHandleCurrentDevice()
Calls getBlasHandle with the current device.
Functions provided by default
-
cudaStream_t getDefaultStreamCurrentDevice()
Calls getDefaultStream with the current device.
-
size_t getTempMemoryAvailableCurrentDevice() const
Calls getTempMemoryAvailable with the current device.
-
GpuMemoryReservation allocMemoryHandle(const AllocRequest &req)
Returns a temporary memory allocation via a RAII object.
-
void syncDefaultStream(int device)
Synchronizes the CPU with respect to the default stream for the given device
-
void syncDefaultStreamCurrentDevice()
Calls syncDefaultStream for the current device.
-
std::vector<cudaStream_t> getAlternateStreamsCurrentDevice()
Calls getAlternateStreams for the current device.
-
cudaStream_t getAsyncCopyStreamCurrentDevice()
Calls getAsyncCopyStream for the current device.
-
virtual ~GpuResources()
-
class GpuResourcesProvider
- #include <GpuResources.h>
Interface for a provider of a shared resources object. This is to avoid interfacing std::shared_ptr to Python
Subclassed by faiss::gpu::GpuResourcesProviderFromInstance, faiss::gpu::StandardGpuResources
Public Functions
-
virtual ~GpuResourcesProvider()
-
virtual std::shared_ptr<GpuResources> getResources() = 0
Returns the shared resources object.
-
virtual ~GpuResourcesProvider()
-
class GpuResourcesProviderFromInstance : public faiss::gpu::GpuResourcesProvider
- #include <GpuResources.h>
A simple wrapper for a GpuResources object to make a GpuResourcesProvider out of it again
Public Functions
-
~GpuResourcesProviderFromInstance() override
-
virtual std::shared_ptr<GpuResources> getResources() override
Returns the shared resources object.
Private Members
-
std::shared_ptr<GpuResources> res_
-
~GpuResourcesProviderFromInstance() override
-
template<typename GpuIndex>
struct IndexWrapper Public Functions
-
IndexWrapper(int numGpus, std::function<std::unique_ptr<GpuIndex>(GpuResourcesProvider*, int)> init)
-
void setNumProbes(size_t nprobe)
-
IndexWrapper(int numGpus, std::function<std::unique_ptr<GpuIndex>(GpuResourcesProvider*, int)> init)
-
class StandardGpuResourcesImpl : public faiss::gpu::GpuResources
- #include <StandardGpuResources.h>
Standard implementation of the GpuResources object that provides for a temporary memory manager
Public Functions
-
StandardGpuResourcesImpl()
-
~StandardGpuResourcesImpl() override
-
void noTempMemory()
Disable allocation of temporary memory; all temporary memory requests will call cudaMalloc / cudaFree at the point of use
-
void setTempMemory(size_t size)
Specify that we wish to use a certain fixed size of memory on all devices as temporary memory. This is the upper bound for the GPU memory that we will reserve. We will never go above 1.5 GiB on any GPU; smaller GPUs (with <= 4 GiB or <= 8 GiB) will use less memory than that. To avoid any temporary memory allocation, pass 0.
-
void setPinnedMemory(size_t size)
Set amount of pinned memory to allocate, for async GPU <-> CPU transfers
-
virtual void setDefaultStream(int device, cudaStream_t stream) override
Called to change the stream for work ordering. We do not own
stream; i.e., it will not be destroyed when the GpuResources object gets cleaned up. We are guaranteed that all Faiss GPU work is ordered with respect to this stream upon exit from an index or other Faiss GPU call.
-
void revertDefaultStream(int device)
Revert the default stream to the original stream managed by this resources object, in case someone called
setDefaultStream.
-
virtual cudaStream_t getDefaultStream(int device) override
Returns the stream for the given device on which all Faiss GPU work is ordered. We are guaranteed that all Faiss GPU work is ordered with respect to this stream upon exit from an index or other Faiss GPU call.
-
void setDefaultNullStreamAllDevices()
Called to change the work ordering streams to the null stream for all devices
-
void setLogMemoryAllocations(bool enable)
If enabled, will print every GPU memory allocation and deallocation to standard output
-
virtual void initializeForDevice(int device) override
Internal system calls.
Initialize resources for this device
-
virtual cublasHandle_t getBlasHandle(int device) override
Returns the cuBLAS handle that we use for the given device.
-
virtual std::vector<cudaStream_t> getAlternateStreams(int device) override
Returns the set of alternative streams that we use for the given device.
-
virtual void *allocMemory(const AllocRequest &req) override
Allocate non-temporary GPU memory.
-
virtual void deallocMemory(int device, void *in) override
Returns a previous allocation.
-
virtual size_t getTempMemoryAvailable(int device) const override
For MemorySpace::Temporary, how much space is immediately available without cudaMalloc allocation?
-
std::map<int, std::map<std::string, std::pair<int, size_t>>> getMemoryInfo() const
Export a description of memory used for Python.
-
virtual std::pair<void*, size_t> getPinnedMemory() override
Returns the available CPU pinned memory buffer.
-
virtual cudaStream_t getAsyncCopyStream(int device) override
Returns the stream on which we perform async CPU <-> GPU copies.
-
cublasHandle_t getBlasHandleCurrentDevice()
Calls getBlasHandle with the current device.
Functions provided by default
-
cudaStream_t getDefaultStreamCurrentDevice()
Calls getDefaultStream with the current device.
-
size_t getTempMemoryAvailableCurrentDevice() const
Calls getTempMemoryAvailable with the current device.
-
GpuMemoryReservation allocMemoryHandle(const AllocRequest &req)
Returns a temporary memory allocation via a RAII object.
-
void syncDefaultStream(int device)
Synchronizes the CPU with respect to the default stream for the given device
-
void syncDefaultStreamCurrentDevice()
Calls syncDefaultStream for the current device.
-
std::vector<cudaStream_t> getAlternateStreamsCurrentDevice()
Calls getAlternateStreams for the current device.
-
cudaStream_t getAsyncCopyStreamCurrentDevice()
Calls getAsyncCopyStream for the current device.
Protected Functions
-
bool isInitialized(int device) const
Have GPU resources been initialized for this device yet?
Protected Attributes
-
std::unordered_map<int, std::unordered_map<void*, AllocRequest>> allocs_
Set of currently outstanding memory allocations per device device -> (alloc request, allocated ptr)
-
std::unordered_map<int, std::unique_ptr<StackDeviceMemory>> tempMemory_
Temporary memory provider, per each device.
-
std::unordered_map<int, cudaStream_t> defaultStreams_
Our default stream that work is ordered on, one per each device.
-
std::unordered_map<int, cudaStream_t> userDefaultStreams_
This contains particular streams as set by the user for ordering, if any
-
std::unordered_map<int, std::vector<cudaStream_t>> alternateStreams_
Other streams we can use, per each device.
-
std::unordered_map<int, cudaStream_t> asyncCopyStreams_
Async copy stream to use for GPU <-> CPU pinned memory copies.
-
std::unordered_map<int, cublasHandle_t> blasHandles_
cuBLAS handle for each device
-
void *pinnedMemAlloc_
Pinned memory allocation for use with this GPU.
-
size_t pinnedMemAllocSize_
-
size_t tempMemSize_
Another option is to use a specified amount of memory on all devices
-
size_t pinnedMemSize_
Amount of pinned memory we should allocate.
-
bool allocLogging_
Whether or not we log every GPU memory allocation and deallocation.
Protected Static Functions
-
static size_t getDefaultTempMemForGPU(int device, size_t requested)
Adjust the default temporary memory allocation based on the total GPU memory size
-
StandardGpuResourcesImpl()
-
class StandardGpuResources : public faiss::gpu::GpuResourcesProvider
- #include <StandardGpuResources.h>
Default implementation of GpuResources that allocates a cuBLAS stream and 2 streams for use, as well as temporary memory. Internally, the Faiss GPU code uses the instance managed by getResources, but this is the user-facing object that is internally reference counted.
Public Functions
-
StandardGpuResources()
-
~StandardGpuResources() override
-
virtual std::shared_ptr<GpuResources> getResources() override
Returns the shared resources object.
-
void noTempMemory()
Disable allocation of temporary memory; all temporary memory requests will call cudaMalloc / cudaFree at the point of use
-
void setTempMemory(size_t size)
Specify that we wish to use a certain fixed size of memory on all devices as temporary memory. This is the upper bound for the GPU memory that we will reserve. We will never go above 1.5 GiB on any GPU; smaller GPUs (with <= 4 GiB or <= 8 GiB) will use less memory than that. To avoid any temporary memory allocation, pass 0.
-
void setPinnedMemory(size_t size)
Set amount of pinned memory to allocate, for async GPU <-> CPU transfers
-
void setDefaultStream(int device, cudaStream_t stream)
Called to change the stream for work ordering. We do not own
stream; i.e., it will not be destroyed when the GpuResources object gets cleaned up. We are guaranteed that all Faiss GPU work is ordered with respect to this stream upon exit from an index or other Faiss GPU call.
-
void revertDefaultStream(int device)
Revert the default stream to the original stream managed by this resources object, in case someone called
setDefaultStream.
-
void setDefaultNullStreamAllDevices()
Called to change the work ordering streams to the null stream for all devices
-
std::map<int, std::map<std::string, std::pair<int, size_t>>> getMemoryInfo() const
Export a description of memory used for Python.
-
cudaStream_t getDefaultStream(int device)
Returns the current default stream.
-
size_t getTempMemoryAvailable(int device) const
Returns the current amount of temp memory available.
-
void syncDefaultStreamCurrentDevice()
Synchronize our default stream with the CPU.
-
void setLogMemoryAllocations(bool enable)
If enabled, will print every GPU memory allocation and deallocation to standard output
Private Members
-
std::shared_ptr<StandardGpuResourcesImpl> res_
-
StandardGpuResources()
-
class DeviceScope
- #include <DeviceUtils.h>
RAII object to set the current device, and restore the previous device upon destruction
Public Functions
-
explicit DeviceScope(int device)
-
~DeviceScope()
Private Members
-
int prevDevice_
-
explicit DeviceScope(int device)
-
class CublasHandleScope
- #include <DeviceUtils.h>
RAII object to manage a cublasHandle_t.
Public Functions
-
CublasHandleScope()
-
~CublasHandleScope()
-
inline cublasHandle_t get()
Private Members
-
cublasHandle_t blasHandle_
-
CublasHandleScope()
-
class CudaEvent
Public Functions
-
explicit CudaEvent(cudaStream_t stream, bool timer = false)
Creates an event and records it in this stream.
-
CudaEvent(const CudaEvent &event) = delete
-
CudaEvent(CudaEvent &&event) noexcept
-
~CudaEvent()
-
inline cudaEvent_t get()
-
void streamWaitOnEvent(cudaStream_t stream)
Wait on this event in this stream.
-
void cpuWaitOnEvent()
Have the CPU wait for the completion of this event.
Private Members
-
cudaEvent_t event_
-
explicit CudaEvent(cudaStream_t stream, bool timer = false)
-
class StackDeviceMemory
- #include <StackDeviceMemory.h>
Device memory manager that provides temporary memory allocations out of a region of memory, for a single device
Public Functions
-
StackDeviceMemory(GpuResources *res, int device, size_t allocPerDevice)
Allocate a new region of memory that we manage.
-
StackDeviceMemory(int device, void *p, size_t size, bool isOwner)
Manage a region of memory for a particular device, with or without ownership
-
~StackDeviceMemory()
-
int getDevice() const
-
void *allocMemory(cudaStream_t stream, size_t size)
All allocations requested should be a multiple of 16 bytes.
-
void deallocMemory(int device, cudaStream_t, size_t size, void *p)
-
size_t getSizeAvailable() const
-
std::string toString() const
-
struct Range
- #include <StackDeviceMemory.h>
Previous allocation ranges and the streams for which synchronization is required
Public Functions
-
inline Range(char *s, char *e, cudaStream_t str)
Public Members
-
char *start_
-
char *end_
-
cudaStream_t stream_
-
inline Range(char *s, char *e, cudaStream_t str)
-
struct Stack
Public Functions
-
Stack(GpuResources *res, int device, size_t size)
Constructor that allocates memory via cudaMalloc.
-
~Stack()
-
size_t getSizeAvailable() const
Returns how much size is available for an allocation without calling cudaMalloc
-
char *getAlloc(size_t size, cudaStream_t stream)
Obtains an allocation; all allocations are guaranteed to be 16 byte aligned
-
void returnAlloc(char *p, size_t size, cudaStream_t stream)
Returns an allocation.
-
std::string toString() const
Returns the stack state.
Public Members
-
GpuResources *res_
Our GpuResources object.
-
int device_
Device this allocation is on.
-
char *alloc_
Where our temporary memory buffer is allocated; we allocate starting 16 bytes into this
-
size_t allocSize_
Total size of our allocation.
-
char *start_
Our temporary memory region; [start_, end_) is valid.
-
char *end_
-
char *head_
Stack head within [start, end)
-
std::list<Range> lastUsers_
List of previous last users of allocations on our stack, for possible synchronization purposes
-
size_t highWaterMemoryUsed_
What’s the high water mark in terms of memory used from the temporary buffer?
-
Stack(GpuResources *res, int device, size_t size)
-
StackDeviceMemory(GpuResources *res, int device, size_t allocPerDevice)
-
class KernelTimer
- #include <Timer.h>
Utility class for timing execution of a kernel.
Public Functions
-
KernelTimer(cudaStream_t stream = 0)
Constructor starts the timer and adds an event into the current device stream
-
~KernelTimer()
Destructor releases event resources.
-
float elapsedMilliseconds()
Adds a stop event then synchronizes on the stop event to get the actual GPU-side kernel timings for any kernels launched in the current stream. Returns the number of milliseconds elapsed. Can only be called once.
Private Members
-
cudaEvent_t startEvent_
-
cudaEvent_t stopEvent_
-
cudaStream_t stream_
-
bool valid_
-
KernelTimer(cudaStream_t stream = 0)
-
class CpuTimer
- #include <Timer.h>
CPU wallclock elapsed timer.
Public Functions
-
CpuTimer()
Creates and starts a new timer.
-
float elapsedMilliseconds()
Returns elapsed time in milliseconds.
-
CpuTimer()
-
namespace utils
Functions
- template<typename U, typename V> constexpr __host__ __device__ auto divDown (U a, V b) -> decltype(a+b)
- template<typename U, typename V> constexpr __host__ __device__ auto divUp (U a, V b) -> decltype(a+b)
- template<typename U, typename V> constexpr __host__ __device__ auto roundDown (U a, V b) -> decltype(a+b)
- template<typename U, typename V> constexpr __host__ __device__ auto roundUp (U a, V b) -> decltype(a+b)
- template<class T> constexpr __host__ __device__ T pow (T n, T power)
- template<class T> constexpr __host__ __device__ T pow2 (T n)
- template<typename T> constexpr __host__ __device__ int log2 (T n, int p=0)
- template<typename T> constexpr __host__ __device__ bool isPowerOf2 (T v)
- template<typename T> constexpr __host__ __device__ T nextHighestPowerOf2 (T v)
-
enum class DistanceDataType
-
namespace ivflib
Functions
-
void check_compatible_for_merge(const Index *index1, const Index *index2)
check if two indexes have the same parameters and are trained in the same way, otherwise throw.
-
const IndexIVF *extract_index_ivf(const Index *index)
get an IndexIVF from an index. The index may be an IndexIVF or some wrapper class that encloses an IndexIVF
throws an exception if this is not the case.
-
const IndexIVF *try_extract_index_ivf(const Index *index)
same as above but returns nullptr instead of throwing on failure
-
void merge_into(Index *index0, Index *index1, bool shift_ids)
Merge index1 into index0. Works on IndexIVF’s and IndexIVF’s embedded in a IndexPreTransform. On output, the index1 is empty.
- Parameters:
shift_ids – translate the ids from index1 to index0->prev_ntotal
-
void search_and_return_centroids(Index *index, size_t n, const float *xin, long k, float *distances, idx_t *labels, idx_t *query_centroid_ids, idx_t *result_centroid_ids)
-
ArrayInvertedLists *get_invlist_range(const Index *index, long i0, long i1)
Get a subset of inverted lists [i0, i1)
-
void set_invlist_range(Index *index, long i0, long i1, ArrayInvertedLists *src)
Set a subset of inverted lists.
-
void search_with_parameters(const Index *index, idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const IVFSearchParameters *params, size_t *nb_dis = nullptr, double *ms_per_stage = nullptr)
search an IndexIVF, possibly embedded in an IndexPreTransform with given parameters. This is a way to set the nprobe and get statdistics in a thread-safe way.
Optionally returns (if non-nullptr):
nb_dis: number of distances computed
ms_per_stage: [0]: preprocessing time [1]: coarse quantization, [2]: list scanning
-
void range_search_with_parameters(const Index *index, idx_t n, const float *x, float radius, RangeSearchResult *result, const IVFSearchParameters *params, size_t *nb_dis = nullptr, double *ms_per_stage = nullptr)
same as search_with_parameters but for range search
-
IndexIVFResidualQuantizer *ivf_residual_from_quantizer(const ResidualQuantizer&, int nlevel)
Build an IndexIVFResidualQuantizer from an ResidualQuantizer, using the nlevel first components as coarse quantizer and the rest as codes in invlists
-
void ivf_residual_add_from_flat_codes(IndexIVFResidualQuantizer *ivfrq, size_t ncode, const uint8_t *codes, int64_t code_size = -1)
add from codes. NB that the norm component is not used, so the code_size can be provided.
- Parameters:
ivfrq – index to populate with the codes
codes – codes to add, size (ncode, code_size)
code_size – override the ivfrq’s code_size, useful if the norm encoding is different
-
struct SlidingIndexWindow
- #include <IVFlib.h>
A set of IndexIVFs concatenated together in a FIFO fashion. at each “step”, the oldest index slice is removed and a new index is added.
Public Functions
-
SlidingIndexWindow(Index *index)
index should be initially empty and trained
-
void step(const Index *sub_index, bool remove_oldest)
Add one index to the current index and remove the oldest one.
- Parameters:
sub_index – slice to swap in (can be NULL)
remove_oldest – if true, remove the oldest slices
Public Members
-
Index *index
common index that contains the sliding window
-
ArrayInvertedLists *ils
InvertedLists of index.
-
int n_slice
number of slices currently in index
-
size_t nlist
same as index->nlist
-
SlidingIndexWindow(Index *index)
-
void check_compatible_for_merge(const Index *index1, const Index *index2)
-
namespace lsq
-
struct IcmEncoder
Subclassed by faiss::gpu::GpuIcmEncoder
Public Functions
-
explicit IcmEncoder(const LocalSearchQuantizer *lsq)
-
inline virtual ~IcmEncoder()
compute binary terms
-
virtual void set_binary_term()
-
virtual void encode(int32_t *codes, const float *x, std::mt19937 &gen, size_t n, size_t ils_iters) const
Encode vectors given codebooks
- Parameters:
codes – output codes, size n * M
x – vectors to encode, size n * d
gen – random generator
n – number of vectors
ils_iters – number of iterations of iterative local search
-
explicit IcmEncoder(const LocalSearchQuantizer *lsq)
-
struct IcmEncoderFactory
Subclassed by faiss::gpu::GpuIcmEncoderFactory
Public Functions
-
inline virtual IcmEncoder *get(const LocalSearchQuantizer *lsq)
-
inline virtual ~IcmEncoderFactory()
-
inline virtual IcmEncoder *get(const LocalSearchQuantizer *lsq)
-
struct LSQTimer
- #include <LocalSearchQuantizer.h>
A helper struct to count consuming time during training. It is NOT thread-safe.
-
struct LSQTimerScope
-
struct IcmEncoder
-
namespace nndescent
-
struct Neighbor
Public Functions
-
Neighbor() = default
-
inline Neighbor(int id, float distance, bool f)
-
inline bool operator<(const Neighbor &other) const
Public Members
-
int id
-
float distance
-
bool flag
-
Neighbor() = default
-
struct Neighbor
-
namespace nsg
Functions
-
DistanceComputer *storage_distance_computer(const Index *storage)
-
template<class node_t>
struct Graph Public Functions
-
inline Graph(node_t *data, int N, int K)
-
inline Graph(int N, int K)
-
inline Graph(const Graph &g)
-
inline ~Graph()
-
inline node_t at(int i, int j) const
-
inline node_t &at(int i, int j)
Public Members
-
node_t *data
the flattened adjacency matrix, size N-by-K
-
int K
nb of neighbors per node
-
int N
total nb of nodes
-
bool own_fields
the underlying data owned by itself or not
-
inline Graph(node_t *data, int N, int K)
-
DistanceComputer *storage_distance_computer(const Index *storage)
-
namespace quantize_lut
Functions to quantize PQ floating-point Look Up Tables (LUT) to uint8, and biases to uint16. The accumulation is supposed to take place in uint16. The quantization coefficients are float (a, b) such that
The hardest part of the quantization is with multiple LUTs that need to be added up together. In that case, coefficient a has to be chosen so that the sum fits in a uint16 accumulator.original_value = quantized_value * a / b
Functions
-
void round_uint8_per_column(float *tab, size_t n, size_t d, float *a_out = nullptr, float *b_out = nullptr)
-
void round_uint8_per_column_multi(float *tab, size_t m, size_t n, size_t d, float *a_out = nullptr, float *b_out = nullptr)
-
void quantize_LUT_and_bias(size_t nprobe, size_t M, size_t ksub, bool lut_is_3d, const float *LUT, const float *bias, uint8_t *LUTq, size_t M2, uint16_t *biasq, float *a_out = nullptr, float *b_out = nullptr)
LUT quantization to uint8 and bias to uint16.
(nprobe, M, ksub, lut_is_3d) determine the size of the the LUT
LUT input:
2D size (M, ksub): single matrix per probe (lut_is_3d=false)
3D size (nprobe, M, ksub): separate LUT per probe (lut_is_3d=true) bias input:
nullptr: bias is 0
size (nprobe): one bias per probe Output:
LUTq uint8 version of the LUT (M size is rounded up to M2)
biasq (or nullptr): uint16 version of the LUT
a, b: scalars to approximate the true distance
-
void aq_quantize_LUT_and_bias(size_t nprobe, size_t M, size_t ksub, const float *LUT, const float *bias, size_t M_norm, int norm_scale, uint8_t *LUTq, size_t M2, uint16_t *biasq, float *a_out, float *b_out)
-
float aq_estimate_norm_scale(size_t M, size_t ksub, size_t M_norm, const float *LUT)
-
void round_uint8_per_column(float *tab, size_t n, size_t d, float *a_out = nullptr, float *b_out = nullptr)
-
namespace rq_encode_steps
Functions
-
void refine_beam_mp(const ResidualQuantizer &rq, size_t n, size_t beam_size, const float *x, int out_beam_size, int32_t *out_codes, float *out_residuals, float *out_distances, RefineBeamMemoryPool &pool)
-
void refine_beam_LUT_mp(const ResidualQuantizer &rq, size_t n, const float *query_norms, const float *query_cp, int out_beam_size, int32_t *out_codes, float *out_distances, RefineBeamLUTMemoryPool &pool)
-
void compute_codes_add_centroids_mp_lut0(const ResidualQuantizer &rq, const float *x, uint8_t *codes_out, size_t n, const float *centroids, ComputeCodesAddCentroidsLUT0MemoryPool &pool)
-
void compute_codes_add_centroids_mp_lut1(const ResidualQuantizer &rq, const float *x, uint8_t *codes_out, size_t n, const float *centroids, ComputeCodesAddCentroidsLUT1MemoryPool &pool)
-
struct RefineBeamMemoryPool
-
struct RefineBeamLUTMemoryPool
-
struct ComputeCodesAddCentroidsLUT0MemoryPool
Public Members
-
std::vector<int32_t> codes
-
std::vector<float> norms
-
std::vector<float> distances
-
std::vector<float> residuals
-
RefineBeamMemoryPool refine_beam_pool
-
std::vector<int32_t> codes
-
struct ComputeCodesAddCentroidsLUT1MemoryPool
-
void refine_beam_mp(const ResidualQuantizer &rq, size_t n, size_t beam_size, const float *x, int out_beam_size, int32_t *out_codes, float *out_residuals, float *out_distances, RefineBeamMemoryPool &pool)
-
namespace simd_result_handlers
Functions
-
template<class C, bool W, class Consumer, class ...Types>
void dispatch_SIMDResultHanlder_fixedCW(SIMDResultHandler &res, Consumer &consumer, Types... args)
-
template<class C, class Consumer, class ...Types>
void dispatch_SIMDResultHanlder_fixedC(SIMDResultHandler &res, Consumer &consumer, Types... args)
-
template<class Consumer, class ...Types>
void dispatch_SIMDResultHanlder(SIMDResultHandler &res, Consumer &consumer, Types... args)
-
struct DummyResultHandler : public faiss::SIMDResultHandler
- #include <simd_result_handlers.h>
Dummy structure that just computes a chqecksum on results (to avoid the computation to be optimized away)
Public Functions
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void set_block_origin(size_t, size_t) final
set the sub-matrix that is being computed
-
inline ~DummyResultHandler()
Public Members
-
size_t cs = 0
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
-
struct StoreResultHandler : public faiss::SIMDResultHandler
- #include <simd_result_handlers.h>
memorize results in a nq-by-nb matrix.
j0 is the current upper-left block of the matrix
Public Functions
-
inline StoreResultHandler(uint16_t *data, size_t ld)
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
Public Members
-
uint16_t *data
-
size_t ld
-
size_t i0 = 0
-
size_t j0 = 0
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
inline StoreResultHandler(uint16_t *data, size_t ld)
-
template<int NQ, int BB>
struct FixedStorageHandler : public faiss::SIMDResultHandler - #include <simd_result_handlers.h>
stores results in fixed-size matrix.
Public Functions
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
-
template<class OtherResultHandler>
inline void to_other_handler(OtherResultHandler &other) const
-
inline virtual ~FixedStorageHandler()
Public Members
-
simd16uint16 dis[NQ][BB]
-
int i0 = 0
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
-
template<class C, bool with_id_map>
struct ResultHandlerCompare : public faiss::SIMDResultHandlerToFloat - #include <simd_result_handlers.h>
Result handler that compares distances to check if they need to be kept
Subclassed by faiss::simd_result_handlers::RangeHandler< C, false >
Public Types
-
using TI = typename C::TI
Public Functions
-
inline ResultHandlerCompare(size_t nq, size_t ntotal, const IDSelector *sel_in)
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
-
inline void adjust_with_origin(size_t &q, simd16uint16 &d0, simd16uint16 &d1)
-
inline int64_t adjust_id(size_t b, size_t j)
-
inline uint32_t get_lt_mask(uint16_t thr, size_t b, simd16uint16 d0, simd16uint16 d1)
return binary mask of elements below thr in (d0, d1) inverse_test returns elements above
-
inline virtual ~ResultHandlerCompare()
-
inline virtual void begin(const float *norms)
-
inline virtual void end()
-
virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) = 0
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
Public Members
-
bool disable = false
-
int64_t i0 = 0
-
int64_t j0 = 0
-
const IDSelector *sel
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
using TI = typename C::TI
-
template<class C, bool with_id_map = false>
struct SingleResultHandler : public faiss::simd_result_handlers::ResultHandlerCompare<C, false> - #include <simd_result_handlers.h>
Special version for k=1
Public Types
-
using T = typename C::T
-
using TI = typename C::TI
-
using RHC = ResultHandlerCompare<C, with_id_map>
Public Functions
-
inline SingleResultHandler(size_t nq, size_t ntotal, float *dis, int64_t *ids, const IDSelector *sel_in)
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void end()
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
-
inline void adjust_with_origin(size_t &q, simd16uint16 &d0, simd16uint16 &d1)
-
inline int64_t adjust_id(size_t b, size_t j)
-
inline uint32_t get_lt_mask(uint16_t thr, size_t b, simd16uint16 d0, simd16uint16 d1)
return binary mask of elements below thr in (d0, d1) inverse_test returns elements above
-
inline virtual void begin(const float *norms)
Public Members
-
std::vector<int16_t> idis
-
float *dis
-
int64_t *ids
-
bool disable
-
int64_t i0
-
int64_t j0
-
const IDSelector *sel
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
using T = typename C::T
-
template<class C, bool with_id_map = false>
struct HeapHandler : public faiss::simd_result_handlers::ResultHandlerCompare<C, false> - #include <simd_result_handlers.h>
Structure that collects results in a min- or max-heap
Public Types
-
using T = typename C::T
-
using TI = typename C::TI
-
using RHC = ResultHandlerCompare<C, with_id_map>
Public Functions
-
inline HeapHandler(size_t nq, size_t ntotal, int64_t k, float *dis, int64_t *ids, const IDSelector *sel_in)
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void end() override
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
-
inline void adjust_with_origin(size_t &q, simd16uint16 &d0, simd16uint16 &d1)
-
inline int64_t adjust_id(size_t b, size_t j)
-
inline uint32_t get_lt_mask(uint16_t thr, size_t b, simd16uint16 d0, simd16uint16 d1)
return binary mask of elements below thr in (d0, d1) inverse_test returns elements above
-
inline virtual void begin(const float *norms)
Public Members
-
std::vector<uint16_t> idis
-
float *dis
-
int64_t *ids
-
int64_t k
-
bool disable
-
int64_t i0
-
int64_t j0
-
const IDSelector *sel
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
using T = typename C::T
-
template<class C, bool with_id_map = false>
struct ReservoirHandler : public faiss::simd_result_handlers::ResultHandlerCompare<C, false> - #include <simd_result_handlers.h>
Simple top-N implementation using a reservoir.
Results are stored when they are below the threshold until the capacity is reached. Then a partition sort is used to update the threshold. Handler built from several ReservoirTopN (one per query)
Public Types
-
using T = typename C::T
-
using TI = typename C::TI
-
using RHC = ResultHandlerCompare<C, with_id_map>
Public Functions
-
inline ReservoirHandler(size_t nq, size_t ntotal, size_t k, size_t cap, float *dis, int64_t *ids, const IDSelector *sel_in)
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void end() override
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
-
inline void adjust_with_origin(size_t &q, simd16uint16 &d0, simd16uint16 &d1)
-
inline int64_t adjust_id(size_t b, size_t j)
-
inline uint32_t get_lt_mask(uint16_t thr, size_t b, simd16uint16 d0, simd16uint16 d1)
return binary mask of elements below thr in (d0, d1) inverse_test returns elements above
-
inline virtual void begin(const float *norms)
Public Members
-
size_t capacity
-
float *dis
-
int64_t *ids
-
AlignedTable<T> all_vals
-
std::vector<ReservoirTopN<C>> reservoirs
-
bool disable
-
int64_t i0
-
int64_t j0
-
const IDSelector *sel
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
using T = typename C::T
-
template<class C, bool with_id_map = false>
struct RangeHandler : public faiss::simd_result_handlers::ResultHandlerCompare<C, false> - #include <simd_result_handlers.h>
Result hanlder for range search. The difficulty is that the range distances have to be scaled using the scaler.
Public Types
-
using T = typename C::T
-
using TI = typename C::TI
-
using RHC = ResultHandlerCompare<C, with_id_map>
Public Functions
-
inline RangeHandler(RangeSearchResult &rres, float radius, size_t ntotal, const IDSelector *sel_in)
-
inline virtual void begin(const float *norms) override
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void end() override
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
-
inline void adjust_with_origin(size_t &q, simd16uint16 &d0, simd16uint16 &d1)
-
inline int64_t adjust_id(size_t b, size_t j)
-
inline uint32_t get_lt_mask(uint16_t thr, size_t b, simd16uint16 d0, simd16uint16 d1)
return binary mask of elements below thr in (d0, d1) inverse_test returns elements above
Public Members
-
RangeSearchResult &rres
-
float radius
-
std::vector<uint16_t> thresholds
-
std::vector<size_t> n_per_query
-
size_t q0 = 0
-
bool disable
-
int64_t i0
-
int64_t j0
-
const IDSelector *sel
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
struct Triplet
-
using T = typename C::T
-
template<class C, bool with_id_map = false>
struct PartialRangeHandler : public faiss::simd_result_handlers::RangeHandler<C, false> Public Types
-
using T = typename C::T
-
using TI = typename C::TI
-
using RHC = RangeHandler<C, with_id_map>
Public Functions
-
inline PartialRangeHandler(RangeSearchPartialResult &pres, float radius, size_t ntotal, size_t q0, size_t q1, const IDSelector *sel_in)
-
inline void shift_n_per_query()
-
inline virtual void end() override
-
inline virtual void begin(const float *norms) override
-
inline virtual void handle(size_t q, size_t b, simd16uint16 d0, simd16uint16 d1) final
called when 32 distances are computed and provided in two simd16uint16. (q, b) indicate which entry it is in the block.
-
inline virtual void set_block_origin(size_t i0_in, size_t j0_in) final
set the sub-matrix that is being computed
-
inline void adjust_with_origin(size_t &q, simd16uint16 &d0, simd16uint16 &d1)
-
inline int64_t adjust_id(size_t b, size_t j)
-
inline uint32_t get_lt_mask(uint16_t thr, size_t b, simd16uint16 d0, simd16uint16 d1)
return binary mask of elements below thr in (d0, d1) inverse_test returns elements above
Public Members
-
RangeSearchPartialResult &pres
-
size_t q0
-
std::vector<Triplet> triplets
-
std::vector<size_t> n_per_query
-
RangeSearchResult &rres
-
float radius
-
std::vector<uint16_t> thresholds
-
bool disable = false
-
int64_t i0 = 0
-
int64_t j0 = 0
-
const IDSelector *sel
-
size_t nq
-
size_t ntotal
-
const idx_t *id_map = nullptr
these fields are used mainly for the IVF variants (with_id_map=true)
-
const int *q_map = nullptr
-
const uint16_t *dbias = nullptr
-
const float *normalizers = nullptr
-
bool is_CMax = false
-
uint8_t sizeof_ids = 0
-
bool with_fields = false
-
using T = typename C::T
-
template<class C, bool W, class Consumer, class ...Types>
-
using IndexIDMap = IndexIDMapTemplate<Index>