File AutoTune.h¶
-
namespace faiss
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.
Throughout the library, vectors are provided as float * pointers. Most algorithms can be optimized when several vectors are processed (added/searched) together in a batch. In this case, they are passed in as a matrix. When n vectors of size d are provided as float * x, component j of vector i is
x[ i * d + j ]
where 0 <= i < n and 0 <= j < d. In other words, matrices are always compact. When specifying the size of the matrix, we call it an n*d matrix, which implies a row-major storage.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. I/O functions can read/write to a filename, a file handle or to an object that abstracts the medium.
The read functions return objects that should be deallocated with delete. All references within these objectes are owned by the object.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Definition of inverted lists + a few common classes that implement the interface.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Since IVF (inverted file) indexes are of so much use for large-scale use cases, we group a few functions related to them in this small library. Most functions work both on IndexIVFs and IndexIVFs embedded within an IndexPreTransform.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. In this file are the implementations of extra metrics beyond L2 and inner product
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Defines a few objects that apply transformations to a set of vectors Often these are pre-processing steps.
-
struct AutoTuneCriterion¶
- #include <AutoTune.h>
Evaluation criterion. Returns a performance measure in [0,1], higher is better.
Subclassed by faiss::IntersectionCriterion, faiss::OneRecallAtRCriterion
Public Functions
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)¶
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const = 0¶
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline virtual ~AutoTuneCriterion()¶
Public Members
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)¶
-
struct OneRecallAtRCriterion : public faiss::AutoTuneCriterion¶
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override¶
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~OneRecallAtRCriterion() override¶
Public Members
-
virtual double evaluate(const float *D, const idx_t *I) const override¶
-
struct IntersectionCriterion : public faiss::AutoTuneCriterion¶
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override¶
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~IntersectionCriterion() override¶
Public Members
-
virtual double evaluate(const float *D, const idx_t *I) const override¶
-
struct OperatingPoint¶
- #include <AutoTune.h>
Maintains a list of experimental results. Each operating point is a (perf, t, key) triplet, where higher perf and lower t is better. The key field is an arbitrary identifier for the operating point.
Includes primitives to extract the Pareto-optimal operating points in the (perf, t) space.
-
struct OperatingPoints¶
Public Functions
-
OperatingPoints()¶
-
int merge_with(const OperatingPoints &other, const std::string &prefix = "")¶
add operating points from other to this, with a prefix to the keys
-
void clear()¶
-
bool add(double perf, double t, const std::string &key, size_t cno = 0)¶
add a performance measure. Return whether it is an optimal point
-
double t_for_perf(double perf) const¶
get time required to obtain a given performance measure
-
void display(bool only_optimal = true) const¶
easy-to-read output
-
void all_to_gnuplot(const char *fname) const¶
output to a format easy to digest by gnuplot
-
void optimal_to_gnuplot(const char *fname) const¶
Public Members
-
std::vector<OperatingPoint> all_pts¶
all operating points
-
std::vector<OperatingPoint> optimal_pts¶
optimal operating points, sorted by perf
-
OperatingPoints()¶
-
struct ParameterRange¶
- #include <AutoTune.h>
possible values of a parameter, sorted from least to most expensive/accurate
-
struct ParameterSpace¶
- #include <AutoTune.h>
Uses a-priori knowledge on the Faiss indexes to extract tunable parameters.
Subclassed by faiss::gpu::GpuParameterSpace
Public Functions
-
ParameterSpace()¶
-
size_t n_combinations() const¶
nb of combinations, = product of values sizes
-
bool combination_ge(size_t c1, size_t c2) const¶
returns whether combinations c1 >= c2 in the tuple sense
-
void display() const¶
print a description on stdout
-
ParameterRange &add_range(const std::string &name)¶
add a new parameter (or return it if it exists)
-
void set_index_parameters(Index *index, size_t cno) const¶
set a combination of parameters on an index
-
void set_index_parameters(Index *index, const char *param_string) const¶
set a combination of parameters described by a string
-
virtual void set_index_parameter(Index *index, const std::string &name, double val) const¶
set one of the parameters, returns whether setting was successful
-
void update_bounds(size_t cno, const OperatingPoint &op, double *upper_bound_perf, double *lower_bound_t) const¶
find an upper bound on the performance and a lower bound on t for configuration cno given another operating point op
-
void explore(Index *index, size_t nq, const float *xq, const AutoTuneCriterion &crit, OperatingPoints *ops) const¶
explore operating points
- Parameters:
index – index to run on
xq – query vectors (size nq * index.d)
crit – selection criterion
ops – resulting operating points
-
inline virtual ~ParameterSpace()¶
Public Members
-
std::vector<ParameterRange> parameter_ranges¶
all tunable parameters
-
int verbose¶
verbosity during exploration
-
int n_experiments¶
nb of experiments during optimization (0 = try all combinations)
-
size_t batchsize¶
maximum number of queries to submit at a time.
-
bool thread_over_batches¶
use multithreading over batches (useful to benchmark independent single-searches)
-
double min_test_duration¶
run tests several times until they reach at least this duration (to avoid jittering in MT mode)
-
ParameterSpace()¶
-
struct AutoTuneCriterion¶