maix0 774f374965 update: make something that collapses tokens into one if it can		2024-09-26 22:20:24 +02:00
..
.gitignore	update: added python version of the tokenizer	2024-09-26 16:51:01 +02:00
app.py	update: make something that collapses tokens into one if it can	2024-09-26 22:20:24 +02:00
collapse.py	update: make something that collapses tokens into one if it can	2024-09-26 22:20:24 +02:00
concat.py	update: make something that collapses tokens into one if it can	2024-09-26 22:20:24 +02:00
flake.lock	update: added python version of the tokenizer	2024-09-26 16:51:01 +02:00
flake.nix	update: added python version of the tokenizer	2024-09-26 16:51:01 +02:00
str_to_token.py	update: make something that collapses tokens into one if it can	2024-09-26 22:20:24 +02:00
ttoken.py	update: make something that collapses tokens into one if it can	2024-09-26 22:20:24 +02:00