File VectorTransform.h
-
namespace faiss
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.
Throughout the library, vectors are provided as float * pointers. Most algorithms can be optimized when several vectors are processed (added/searched) together in a batch. In this case, they are passed in as a matrix. When n vectors of size d are provided as float * x, component j of vector i is
x[ i * d + j ]
where 0 <= i < n and 0 <= j < d. In other words, matrices are always compact. When specifying the size of the matrix, we call it an n*d matrix, which implies a row-major storage.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. I/O functions can read/write to a filename, a file handle or to an object that abstracts the medium.
The read functions return objects that should be deallocated with delete. All references within these objectes are owned by the object.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Definition of inverted lists + a few common classes that implement the interface.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Since IVF (inverted file) indexes are of so much use for large-scale use cases, we group a few functions related to them in this small library. Most functions work both on IndexIVFs and IndexIVFs embedded within an IndexPreTransform.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. In this file are the implementations of extra metrics beyond L2 and inner product
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Defines a few objects that apply transformations to a set of vectors Often these are pre-processing steps.
-
struct CenteringTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
Subtract the mean of each component from the vectors.
Public Functions
-
explicit CenteringTransform(int d = 0)
Public Members
-
std::vector<float> mean
Mean, size d_in = d_out.
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit CenteringTransform(int d = 0)
-
struct ITQMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
ITQ implementation from
Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval,
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, Florent Perronnin, PAMI’12.
Public Functions
-
explicit ITQMatrix(int d = 0)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
Public Members
-
int max_iter
-
int seed
-
std::vector<double> init_rotation
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit ITQMatrix(int d = 0)
-
struct ITQTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
The full ITQ transform, including normalizations and PCA transformation
Public Functions
-
explicit ITQTransform(int d_in = 0, int d_out = 0, bool do_pca = false)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
explicit ITQTransform(int d_in = 0, int d_out = 0, bool do_pca = false)
-
struct LinearTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
Generic linear transformation, with bias term applied on output y = A * x + b
Subclassed by faiss::ITQMatrix, faiss::OPQMatrix, faiss::PCAMatrix, faiss::RandomRotationMatrix
Public Functions
-
explicit LinearTransform(int d_in = 0, int d_out = 0, bool have_bias = false)
both d_in > d_out and d_out < d_in are supported
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
inline ~LinearTransform() override
Public Members
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit LinearTransform(int d_in = 0, int d_out = 0, bool have_bias = false)
-
struct NormalizationTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
per-vector normalization
Public Functions
-
explicit NormalizationTransform(int d, float norm = 2.0)
-
NormalizationTransform()
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
Identity transform since norm is not revertible.
Public Members
-
float norm
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit NormalizationTransform(int d, float norm = 2.0)
-
struct OPQMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Applies a rotation to align the dimensions with a PQ to minimize the reconstruction error. Can be used before an IndexPQ or an IndexIVFPQ. The method is the non-parametric version described in:
“Optimized Product Quantization for Approximate Nearest Neighbor Search” Tiezheng Ge, Kaiming He, Qifa Ke, Jian Sun, CVPR’13
Public Functions
-
explicit OPQMatrix(int d = 0, int M = 1, int d2 = -1)
if d2 != -1, output vectors of this dimension
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
Public Members
-
int M
nb of subquantizers
-
int niter
Number of outer training iterations.
-
int niter_pq
Number of training iterations for the PQ.
-
int niter_pq_0
same, for the first outer iteration
-
size_t max_train_points
if there are too many training points, resample
-
bool verbose
-
ProductQuantizer *pq
if non-NULL, use this product quantizer for training should be constructed with (d_out, M, _)
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit OPQMatrix(int d = 0, int M = 1, int d2 = -1)
-
struct PCAMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Applies a principal component analysis on a set of vectors, with optionally whitening and random rotation.
Public Functions
-
explicit PCAMatrix(int d_in = 0, int d_out = 0, float eigen_power = 0, bool random_rotation = false)
-
virtual void train(idx_t n, const float *x) override
train on n vectors. If n < d_in then the eigenvector matrix will be completed with 0s
-
void prepare_Ab()
called after mean, PCAMat and eigenvalues are computed
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
Public Members
-
float eigen_power
after transformation the components are multiplied by eigenvalues^eigen_power
=0: no whitening =-0.5: full whitening
-
float epsilon
value added to eigenvalues to avoid division by 0 when whitening
-
bool random_rotation
random rotation after PCA
-
size_t max_points_per_d
ratio between # training vectors and dimension
-
int balanced_bins
try to distribute output eigenvectors in this many bins
-
std::vector<float> mean
Mean, size d_in.
-
std::vector<float> eigenvalues
eigenvalues of covariance matrix (= squared singular values)
-
std::vector<float> PCAMat
PCA matrix, size d_in * d_in.
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit PCAMatrix(int d_in = 0, int d_out = 0, float eigen_power = 0, bool random_rotation = false)
-
struct RandomRotationMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Randomly rotate a set of vectors.
Public Functions
-
inline RandomRotationMatrix(int d_in, int d_out)
both d_in > d_out and d_out < d_in are supported
-
void init(int seed)
must be called before the transform is used
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters
n – nb of training vectors
x – training vecors, size n * d
-
inline RandomRotationMatrix()
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
Public Members
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
inline RandomRotationMatrix(int d_in, int d_out)
-
struct RemapDimensionsTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
remap dimensions for intput vectors, possibly inserting 0s strictly speaking this is also a linear transform but we don’t want to compute it with matrix multiplies
Public Functions
-
RemapDimensionsTransform(int d_in, int d_out, const int *map)
-
RemapDimensionsTransform(int d_in, int d_out, bool uniform = true)
remap input to output, skipping or inserting dimensions as needed if uniform: distribute dimensions uniformly otherwise just take the d_out first ones.
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
reverse transform correct only when the mapping is a permutation
-
inline RemapDimensionsTransform()
Public Members
-
std::vector<int> map
map from output dimension to input, size d_out -1 -> set output to 0
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
RemapDimensionsTransform(int d_in, int d_out, const int *map)
-
struct VectorTransform
- #include <VectorTransform.h>
Any transformation applied on a set of vectors
Subclassed by faiss::CenteringTransform, faiss::ITQTransform, faiss::LinearTransform, faiss::NormalizationTransform, faiss::RemapDimensionsTransform
Public Functions
-
inline explicit VectorTransform(int d_in = 0, int d_out = 0)
! output dimension
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters
x – size n * d_in
- Returns
size n * d_out
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const = 0
same as apply, but result is pre-allocated
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const
reverse transformation. May not be implemented or may return approximate result
-
inline virtual ~VectorTransform()
Public Members
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
inline explicit VectorTransform(int d_in = 0, int d_out = 0)
-
struct CenteringTransform : public faiss::VectorTransform