はてなブックマーク

In the context of building LLM-related applications, chunking is the process of breaking down large pieces of text into smaller segments. It’s an essential technique that helps optimize the relevance of the content we get back from a vector database once we use the LLM to embed content. In this blog post, we’ll explore if and how it helps improve efficiency and accuracy in LLM-related applications

What is a Vector Database?A vector database indexes and stores vector embeddings for fast retrieval and similarity search, with capabilities like CRUD operations, metadata filtering, horizontal scaling, and serverless. We’re in the midst of the AI revolution. It’s upending any industry it touches, promising great innovations - but it also introduces new challenges. Efficient data processing has be

The LangChain library empowers developers to create intelligent applications using large language models. It’s revolutionizing industries and technology, transforming our every interaction with technology.

Create an account and your first index in 30 seconds, then upload a few vector embeddings from any model… or a few billion. Perform low-latency vector search to retrieve relevant data for search, RAG, recommendation, detection, and other applications. Pinecone is serverless so you never have to worry about managing or scaling the database. from pinecone import Pinecone, ServerlessSpec # Create a s

Hierarchical Navigable Small World (HNSW) graphs are among the top-performing indexes for vector similarity search[1]. HNSW is a hugely popular technology that time and time again produces state-of-the-art performance with super fast search speeds and fantastic recall. Yet despite being a popular and robust algorithm for approximate nearest neighbors (ANN) searches, understanding how it works is f

Vector similarity search is a game-changer in the world of search. It allows us to efficiently search a huge range of media, from GIFs to articles — with incredible accuracy in sub-second timescales for billion+ size datasets. One of the key components to efficient search is flexibility. And for that we have a wide range of search indexes available to us — there is no ‘one-size-fits-all’ in simila

Fortunately, it’s a brilliantly simple process to get started with. And in this article, we’ll explore some of the options FAISS provides, how they work, and — most importantly — how Faiss can make our search faster. Check out the video walkthrough here: What is Faiss?Before we get started with any code, many of you will be asking — what is Faiss? Faiss is a library — developed by Facebook AI — th