Full text search with limit seems to be giving incorrect answer in this sample case

[westonpace]

In the below example we do a FTS on two strings. If we don't apply a limit (or set the limit to 2) then we can see the shorter string has a higher score (presumably because a higher percentage of words are contained in the match?). However, if we set the limit to 1 then we get the document with the lower score back.

import lance
import pyarrow as pa
import shutil

shutil.rmtree("/tmp/foo.lance", ignore_errors=True)

tab = pa.table({
    "text": ["this is some text", "this is some other text"]
})
ds = lance.write_dataset(tab, "/tmp/foo.lance")
ds.create_scalar_index("text", index_type="INVERTED")

print("Results with limit(1)")
print("---------------------")
print(ds.to_table(full_text_query="some text", limit=1))

print("Results with limit(2)")
print("---------------------")
print(ds.to_table(full_text_query="some text", limit=2))