Web App
NEWCompare Tokenizer Vocabularies And Special Tokens
Locate tokenizer changes affecting encoded text.
Web Application
Try Webapp
Web ApplicationRuns in your browser without installing software. Browser access uses your Sobolsoft account and its current plan.
Processing takes place in your browser, without uploading your files for processing. Results are saved to your device.
The app's ? Help contains instructions, supported formats and browser limits.
Screenshot
Inspect the interface
Description
What is this software?
Locate tokenizer changes affecting encoded text.
Features
- Compare tokenizer vocabulary IDs, added tokens, special-token declarations and normalization settings.
- Select which tokenizer sections to compare and retain exact before-and-after differences.
Locate tokenizer changes affecting encoded text.
Features
- Compare tokenizer vocabulary IDs, added tokens, special-token declarations and normalization settings.
- Select which tokenizer sections to compare and retain exact before-and-after differences.