• File size measurements show the distribution of size of files.
• Files are classified in four categories based on their size (lines of code): 1-100 (very small files), 101-200 (small files), 201-500 (medium size files), 501-1000 (long files), 1001+(very long files).
• It is a good practice to keep files small. Long files may become "bloaters", code that have increased to such gargantuan proportions that they are hard to work with.

• There are 146 files with 29,051 lines of code.
• 3 very long files (4,247 lines of code)
• 15 long files (10,551 lines of code)
• 25 medium size files (7,147 lines of codeclsfd_ftr_w_mp_ins)
• 28 small files (3,671 lines of code)
• 75 very small files (3,435 lines of code)

There are 18 files with lines longer than 120 characters. In total, there are 54 long lines.

File	# lines	# units	# long lines
pre_tokenizers.rs in bindings/python/src	791	62	11
processors.rs in bindings/python/src	642	48	9
serialization.rs in tokenizers/src/models/bpe	217	6	5
normalizers.rs in bindings/python/src	760	58	5
strip.rs in tokenizers/src/normalizers	133	6	2
from_pretrained.rs in tokenizers/src/utils	58	1	2
mod.rs in tokenizers/src/pre_tokenizers	258	2	2
metaspace.rs in tokenizers/src/pre_tokenizers	331	10	2
mod.rs in tokenizers/src/models	59	2	2
mod.rs in tokenizers/src/tokenizer	1185	31	2
trainers.rs in bindings/python/src	836	59	2
decoders.rs in bindings/python/src	577	53	2
models.rs in bindings/python/src	785	58	2
tokenizer.rs in bindings/python/src	1060	74	2
encoding.rs in tokenizers/src/tokenizer	780	10	1
template.rs in tokenizers/src/processors	854	35	1
sentencepiece_unigram.py in bindings/python/py_src/tokenizers/implementations	120	4	1
convert.py in bindings/python/scripts	342	32	1