update: added python version of the tokenizer

This commit is contained in:
maix0 2024-09-26 16:51:01 +02:00
parent 40a84e8248
commit 06c2d19097
6 changed files with 339 additions and 0 deletions

1
parser/token.py/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
__pycache__