'how to use bm25 in spark

I have more than 1 million documents to search, and more than 100,000 keywords. Each keyword needs to search 10 most similar documents in the offline way. So how to use BM25 in spark2, thanks ahead.

apache-sparklucene


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions

An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene54' does not exist

package org.apache.commons.io does not exist error

Error while executing data provider - Pimcore DynamicSearch

Use SortedSetDocValuesField to sort multiple fields - Error: unexpected docvalues type SORTED_SET for field

How to use a Scala class inside Pyspark

Spark streaming jdbc read the stream as and when data comes - Data source jdbc does not support streamed reading

How to apply the describe function after grouping a PySpark DataFrame?

How do you create merge_asof functionality in PySpark?

Getting java.lang.NoSuchMethodError: org.yaml.snakeyaml.Yaml.<init> while running spark based spring boot application

Google cloud dataproc cluster created with an environment.yaml with a jupyter resource but environment not available as a jupyter kernel

How to Override log4j with log4j2 version to resolve "SocketServer class vulnerable to deserialization" for apache-core_2.12 version

Spark Streaming "ERROR JobScheduler: error in job generator"

How to correctly import pyspark.sql.functions?

How do we do Spark Dataframe testing using JUnit?

Need to load data from Hadoop to Druid after applying transformations. If I use Spark, can we load data from Spark RDD or dataframe to Druid directly?