Add smoothing constant to IDF formula in BM25 to prevent negative scores (#5696)

Co-authored-by: prayas7102 <prayas.prithvirajpratap7@example.com>
Co-authored-by: Alex Klymenko <alexanderklmn@gmail.com>
This commit is contained in:
Prayas Kumar
2024-10-10 13:09:22 +05:30
committed by GitHub
parent d4fff30eaa
commit 90d20b3a43
2 changed files with 8 additions and 6 deletions

View File

@ -215,6 +215,6 @@ public final class BM25InvertedIndex {
*/
private double computeIDF(int docFrequency) {
// Total number of documents in the index
return Math.log((totalDocuments - docFrequency + 0.5) / (docFrequency + 0.5));
return Math.log((totalDocuments - docFrequency + 0.5) / (docFrequency + 0.5) + 1);
}
}