Skip to content

[Fix] Truncate texts longer than the max_tokens

Romane Gallienne requested to merge fix-text-length-pretrained-detectors into main

What does this MR do and why

Enable truncation when the text to detect is longer than the Detector max_tokens. Not hard-coded as the tokenizer enable to do this automatically.

MR acceptance checklist

Please evaluate this MR against this checklist :

  • Linting and typing are OK,
  • Licenses are OK (licensecheck + integration with reuse with the proper licenses),
  • You have tested this MR locally,
  • You have considered performance issues,
  • You have considered availability and reliability risks,
  • You have updated documentation if necessary.

How to set up and validate locally

Numbered steps to set up and validate the change.

Edited by Romane Gallienne

Merge request reports

Loading