[Fix] Truncate texts longer than the max_tokens
What does this MR do and why
Enable truncation when the text to detect is longer than the Detector max_tokens
. Not hard-coded as the tokenizer enable to do this automatically.
MR acceptance checklist
Please evaluate this MR against this checklist :
-
Linting and typing are OK, -
Licenses are OK (licensecheck + integration with reuse with the proper licenses), -
You have tested this MR locally, -
You have considered performance issues, -
You have considered availability and reliability risks, -
You have updated documentation if necessary.
How to set up and validate locally
Numbered steps to set up and validate the change.
Edited by Romane Gallienne