4 commits

Author	SHA1	Message	Date
maix0	774f374965	update: make something that collapses tokens into one if it can	2024-09-26 22:20:24 +02:00
maix0	c284eb3786	update: commented the python code so it is more readable	2024-09-26 17:50:57 +02:00
maix0	285104a19a	update: pushed more readable tokens	2024-09-26 17:37:57 +02:00
maix0	06c2d19097	update: added python version of the tokenizer	2024-09-26 16:51:01 +02:00