Fixed Length Pre-Tokenizer

[jonvet]

Introduces a pre-tokenizer to split text in fixed length chunks (closes #1697).
The method pre_tokenize could be more made more concise by creating a vector with indices first like so

    let mut splits = Vec::new();

    for chunk in char_positions.chunks(self.length) {
        let start = chunk.first().map(|(i, _)| *i).unwrap_or(0);
        let end = chunk.last().map(|(i, c)| i + c.len_utf8()).unwrap_or(text.len());
        splits.push(normalized.slice(Range::Normalized(start..end))
            .ok_or("Failed to slice normalized text")?);
    }

but that would take a bit more memory, so I went for my approach instead.

[Narsil]

Thanks for this. The code looks working, but I think it could be simplified quite a lot.

Is there any source/paper for trying to do fixed sized chunking ? Before adding anything to the library usually we try to make sure it's used in the wild and would benefits actual users of models (not necessarily researchers exploring new ideas, for this they can try out your branch or create their own pre_tokenizer directly in Python).

[jonvet]

You're right, I simplified it along the lines of my initial comment.

I also asked the author of the issue whether this is a common approach in the literature or not (I'm not aware of it either). Should have probably clarified this before jumping on it ;)

[jonvet]

according to the author it's used in DNA Transformer

[ArthurZucker]

Same as my colleague! would be nice if we can get a reference to the paper it was used in the documentation of the class! (like a arxiv link)
Otherwise we can also keep this issue open an let the community upvote ! if it get's traction we merge 🤗

[ArthurZucker]

we'd also want to make sure that it does it's job as a pretokenizer! so testing with the same string, that it splits in 5 then 10!