Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Website
Tasks
HuggingChat
Collections
Languages
Organizations
Community
Blog
Posts
Daily Papers
Learn
Discord
Forum
GitHub
Solutions
Team & Enterprise
Hugging Face PRO
Enterprise Support
Inference Providers
Inference Endpoints
Storage Buckets
Log In
Sign Up
ggunio
/
B2NL-IntelligentTokenizer-v6.2.1
like
0
PyTorch
Safetensors
flores200
5 languages
b2nl_tokenizer
tokenizer
byte-to-nl
compression
embeddings
semantic-boundaries
Eval Results (legacy)
License:
apache-2.0
Model card
Files
Files and versions
xet
Community
Copy to bucket
new
main
B2NL-IntelligentTokenizer-v6.2.1
/
core
126 kB
Ctrl+K
Ctrl+K
1 contributor
History:
1 commit
ggunio
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
ffbd655
verified
8 months ago
decoder.py
Safe
19.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
8 months ago
encoder.py
Safe
24.1 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
8 months ago
intelligent_loss.py
Safe
21.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
8 months ago
scheduler.py
Safe
23.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
8 months ago
tokenizer.py
Safe
16.8 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
8 months ago
unified_model.py
Safe
21.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
8 months ago
