replace or remove whoosh

[sbrunato]

Whoosh is not actively maintained and sometimes causes errors when several processes try to write in index at the same time.

It may be a good option to directly work on the product-types configuration dictionary, or try another library.

Check how TinyDB could replace whoosh in eodag. Usage example:

import re
from eodag import EODataAccessGateway
from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage


dag = EODataAccessGateway()
pts = dag.list_product_types()

db = TinyDB(storage=MemoryStorage)
db.insert_multiple(pts)

ProductType = Query()

filtered_s2 = db.search(ProductType.ID.matches(r'S2_*'))
filtered_keywords = db.search(ProductType.keywords.matches(r'.*sentinel.*', flags=re.IGNORECASE))

[sbrunato]

Trying to filter on 1676 product-types using whoosh, tinydb, or built-in python list comprehension using free text search:

# whoosh: elapsed time 0.010s
dag.guess_product_type("(Sentinel OR Landsat) AND OPTICAL")

# tinydb: elapsed time 0.013s
[
    pt["ID"] for pt in db.search(
        lambda record: (
            any("Sentinel" in str(value) for value in record.values()) or 
            any("Landsat" in str(value) for value in record.values())
        ) and any("OPTICAL" in str(value) for value in record.values())
    )
]

# built-in python: elapsed time 0.013s
[
        k for k, v in dag.product_types_config.items() 
        if (
            any("Sentinel" in str(x) for x in v.values()) or 
            any("Landsat" in str(x) for x in v.values())
        ) and any("OPTICAL" in str(x) for x in v.values())
 ]

Built-in python list comprehension performance is equivalent to whoosh and tinydb. It avoids dependencies and whoosh build index.
But it needs to efficiently translate free-text-search to appropriate python filtering.