Another aspect to consider is the size and density of the vector dataset.

• IVFFlat is better for large, dense datasets.
• HNSW is better where initial data is sparse or data accumulates gradually (our use case).

HNSW creates a multi-layered graph for the vectors. A search query navigates the layers (or zoom levels) to find the nearest neighbours. This requires longer indexing times and more memory but outperforms IVFFlat in speed-recall metrics.

IVFFlat divides the vectors in clusters. While querying, the clusters closest to the search vector are chosen. It's a straight-forward implementation promising faster indexing times and less memory usage.

I'm using pgvector with postgres for Exhibit AI. I now have to choose an indexing method between IVFFlat and HNSW for the VectorField.

New blog post: Announcing Exhibit AI

3. Then they imitate, remix and repeat.
4. Finally, they share their version with the intended tribe.

Depending on how remarkable the work is, the tribe loves it, hates it or worse, ignores it.

Creators try to reverse engineer this.

1. They choose a tribe — movie lovers that love mindless action (or mind-bending sci-fi thrillers).
2. Then they breakdown and analyse things that the tribe loves already.

Tribes form on the extreme ends of the spectrum.

You talk about something passionately when you love it, or hate it.

Some items immediately grab your attention (without your effort). They are on the far ends of the spectrum.

With repeat exposure or closer inspection, your emotion grows and solidifies.

Imagine a song from Ilaiyaraja or ARR.

In the spectrum of taste (from good to bad), items in the middle are mostly neutral and invisible.

Upon deliberate observation, some may shift left (good) or right (bad).