Struct faiss::IndexShardsIVF

struct IndexShardsIVF : public faiss::IndexShardsTemplate<Index>, public faiss::Level1Quantizer

IndexShards with a common coarse quantizer. All the indexes added should be IndexIVFInterface indexes so that the search_precomputed can be called.

Public Types

using component_t = typename IndexT::component_t

using distance_t = typename IndexT::distance_t

Public Functions

explicit IndexShardsIVF(Index *quantizer, size_t nlist, bool threaded = false, bool successive_ids = true)

void addIndex(Index *index) override

void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override

void train(idx_t n, const component_t *x) override

void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels, const SearchParameters *params = nullptr) const override

inline void add_shard(IndexT *index): Alias for addIndex()

inline void remove_shard(IndexT *index): Alias for removeIndex()

void add(idx_t n, const component_t *x) override: supported only for sub-indices that implement add_with_ids

virtual void syncWithSubIndexes(): Synchronize the top-level index (IndexShards) with data in the sub-indices

virtual void addIndex(IndexT *index): override an index that is managed by ourselves. WARNING: once an index is added, it becomes unsafe to touch it from any other thread than that on which is managing it, until we are shut down. Use runOnIndex to perform work on it instead.

void removeIndex(IndexT *index): Remove an index that is managed by ourselves. This will flush all pending work on that index, and then shut down its managing thread, and will remove the index.

void runOnIndex(std::function<void(int, IndexT*)> f): Run a function on all indices, in the thread that the index is managed in. Function arguments are (index in collection, index pointer)

void runOnIndex(std::function<void(int, const IndexT*)> f) const

void reset() override: faiss::Index API All indices receive the same call

inline int count() const: Returns the number of sub-indices.

inline IndexT *at(size_t i): Returns the i-th sub-index.

inline const IndexT *at(size_t i) const: Returns the i-th sub-index (const version)

void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type): Trains the quantizer and calls train_residual to train sub-quantizers.

size_t coarse_code_size() const: compute the number of bytes required to store list ids

void encode_listno(idx_t list_no, uint8_t *code) const

idx_t decode_listno(const uint8_t *code) const

Public Members

bool successive_ids

bool own_indices = false: Whether or not we are responsible for deleting our contained indices.

Index *quantizer = nullptr: quantizer that maps vectors to inverted lists

size_t nlist = 0: number of inverted lists

char quantizer_trains_alone = 0: = 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer

bool own_fields = false: whether object owns the quantizer

ClusteringParameters cp: to override default clustering params

Index *clustering_index = nullptr: to override index used during clustering

Protected Functions

virtual void onAfterAddIndex(IndexT *index) override: Called just after an index is added.

virtual void onAfterRemoveIndex(IndexT *index) override: Called just after an index is removed.

Protected Attributes

std::vector<std::pair<IndexT*, std::unique_ptr<WorkerThread>>> indices_: Collection of Index instances, with their managing worker thread if any.

bool isThreaded_: Is this index multi-threaded?

Protected Static Functions

static void waitAndHandleFutures(std::vector<std::future<bool>> &v)