Web App
NEWMeasure Dataset Token Counts In Batches
Calculate model-specific token lengths for local datasets.
Web Application
Try Webapp
Web ApplicationRuns in your browser without installing software. Browser access uses your Sobolsoft account and its current plan.
Processing takes place in your browser, without uploading your files for processing. Results are saved to your device.
The app's ? Help contains instructions, supported formats and browser limits.
Screenshot
Inspect the interface
Description
What is this software?
Calculate model-specific token lengths for local datasets.
Features
- Count dataset tokens with the supplied local tokenizer, including declared chat templates and special tokens.
- Flag context-limit overruns and save record counts and a collection summary.
Calculate model-specific token lengths for local datasets.
Features
- Count dataset tokens with the supplied local tokenizer, including declared chat templates and special tokens.
- Flag context-limit overruns and save record counts and a collection summary.