Struct faiss::IndexShardsIVF

struct IndexShardsIVF : public faiss::IndexShardsTemplate<Index>, public faiss::Level1Quantizer

IndexShards with a common coarse quantizer. All the indexes added should be IndexIVFInterface indexes so that the search_precomputed can be called.

Public Types

using component_t = typename IndexT::component_t
using distance_t = typename IndexT::distance_t

Public Functions

explicit IndexShardsIVF(Index *quantizer, size_t nlist, bool threaded = false, bool successive_ids = true)
void addIndex(Index *index) override
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
void train(idx_t n, const component_t *x) override
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override
inline void add_shard(IndexT *index)

Alias for addIndex()

inline void remove_shard(IndexT *index)

Alias for removeIndex()

void add(idx_t n, const component_t *x) override

supported only for sub-indices that implement add_with_ids

virtual void syncWithSubIndexes()

Synchronize the top-level index (IndexShards) with data in the sub-indices

virtual void addIndex(IndexT *index)

override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.

void removeIndex(IndexT *index)

Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.

void runOnIndex(std::function<void(int, IndexT*)> f)

Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)

void runOnIndex(std::function<void(int, const IndexT*)> f) const
void reset() override

faiss::Index API All indices receive the same call

inline int count() const

Returns the number of sub-indices.

inline IndexT *at(size_t i)

Returns the i-th sub-index.

inline const IndexT *at(size_t i) const

Returns the i-th sub-index (const version)

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)

Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const

compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const
idx_t decode_listno(const uint8_t *code) const

Public Members

bool successive_ids
bool own_indices = false

Whether or not we are responsible for deleting our contained indices.

Index *quantizer = nullptr

quantizer that maps vectors to inverted lists

size_t nlist = 0

number of inverted lists

char quantizer_trains_alone = 0

= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer

bool own_fields = false

whether object owns the quantizer

ClusteringParameters cp

to override default clustering params

Index *clustering_index = nullptr

to override index used during clustering

Protected Functions

virtual void onAfterAddIndex(IndexT *index) override

Called just after an index is added.

virtual void onAfterRemoveIndex(IndexT *index) override

Called just after an index is removed.

Protected Attributes

std::vector<std::pair<IndexT*, std::unique_ptr<WorkerThread>>> indices_

Collection of Index instances, with their managing worker thread if any.

bool isThreaded_

Is this index multi-threaded?

Protected Static Functions

static void waitAndHandleFutures(std::vector<std::future<bool>> &v)