Understanding the BM25 full text search algorithm

Nov 19, 2024 BM25, or Best Match 25, is a widely used algorithm for full text search. It is the default in Lucene/Elasticsearch and SQLite, among others. Recently, it has become common to combine full text search and vector similarity search into "hybrid search". I wanted to understand how full text search works, and specifically BM25, so here is my attempt at understanding by re-explaining. Motiv

[misshiki]

“最近では、全文検索とベクトル類似性検索を組み合わせた「ハイブリッド検索」が一般的になっています。私は全文検索、特に BM25 の仕組みを理解したかったので、ここで改めて説明して理解しよう”

[thekoruku]

ずいぶん渋い最適化手法の話だ。tfidfと並んで古典的な手法ですね