The Folkestone projectReciprocal rank fusion: one answer from three lists that disagree
A novel, a five-word question and three rankers, each wrong in a different way.
Kipps, a novel by H. G. Wells, is free to read on Project Gutenberg in an edition dated 1906, and the word Folkestone appears in it 66 times. The learner downloads the text, cuts it at blank lines and keeps every passage of 25 words or more. That leaves 1,138 passages holding 92,961 words. The question for the agent is where the book describes the Leas, the lift, the Harbour and the sea at Folkestone, so the query is five words: folkestone, leas, lift, harbour, sea. One hundred passages contain at least one of them.
Ranker A counts how many times the query words occur. Ranker B divides that count by the length of the passage. Ranker C counts how many different query words are present and uses the raw count only to break ties. Each is a few lines of Python, and each has a weakness. A likes long passages because they have more room for hits. B likes short ones: its winner is a 31-word passage that says "sea" twice and has nothing to do with the town. C cannot tell a passing mention from a description.
How the three rankers placed four passages, and the fused result (k = 60), our Python run| Passage | Rank in A, B, C | Fused score | Fused place |
|---|
| 128 words, 5 hits, the walk "along the Leas to the lift" | 1, 4, 1 | 0.04841 | 1st |
| 107 words, 5 hits, looking down on the Harbour | 2, 2, 2 | 0.04839 | 2nd |
| 105 words, 3 hits, Sunday along the front of the Leas | 5, 15, 4 | 0.04434 | 3rd |
| 31 words, "sea" twice, no mention of the town | 10, 1, 16 | 0.04384 | 4th |
The fusion rule is this: for each passage, add up 1 divided by (60 plus its rank) across the lists, then sort by the total. The first row scores 1/61 + 1/64 + 1/61, which is 0.04841. The three top-ten lists had little in common: A and B shared three passages, and only two passages were in all three. The fused top ten kept nine of A's ten, three of B's and six of C's. B's misleading winner was pushed down to fourth because the other two rankers placed it 10th and 16th. One passage that no single ranker put in its top ten (it was 16th, 21st and 11th) reached ninth place after fusion, because being fairly good on every list counts for a lot.
Two honest limits. Nobody has marked which passages are truly the right answers, so this run shows how fusion behaves, not that it is correct. And the constant 60 comes from the 2009 paper; the learner tries 1 and 1,000 to see what changes and why a middling value is forgiving.
Ages 8 to 11
Three friends each rank ten ice cream flavours. Merge the lists by hand with a points rule and argue about whether it is fair.
Ages 11 to 15
Write ranker A in Python, print its top five passages from Kipps, and spot what it gets wrong.
Ages 15 and up
Build all three rankers, fuse them, and change the constant to watch the order shift.
Sources and what is ours
The text is Project Gutenberg eBook 39162, which Gutenberg distributes free of charge. The fusion formula is from Cormack, Clarke and Buettcher (2009). The passage rule, the query, the three rankers and every rank and score above come from our own run.
