FlyGPT / full-cns /tokenizer_config.json

Whole fly nervous system: 160,514 neurons, 10.4M trainable synapses, best val 1.5957

68282fa verified about 1 month ago

152 Bytes

{
  "tokenizer_class": "PreTrainedTokenizerFast",
  "model_max_length": 10000000,
  "clean_up_tokenization_spaces": false,
  "add_prefix_space": false
}

	{
	"tokenizer_class": "PreTrainedTokenizerFast",
	"model_max_length": 10000000,
	"clean_up_tokenization_spaces": false,
	"add_prefix_space": false
	}