openai / tiktoken
File Change Frequency

File change frequency (churn) shows the distribution of file updates (days with at least one commit).

Overview
File Change Frequency Overall
  • There are 14 files with 1,398 lines of code.
    • 0 files changed more than 100 times (0 lines of code)
    • 0 files changed 51-100 times (0 lines of code)
    • 0 files changed 21-50 times (0 lines of code)
    • 7 files changed 6-20 times (956 lines of code)
    • 7 files changed 1-5 times (442 lines of code)
0% | 0% | 0% | 68% | 31%
Legend:
101+
51-100
21-50
6-20
1-5

explore: grouped by folders | grouped by update frequency | data
Contributors Count Frequency Overall
  • There are 14 files with 1,398 lines of code.
    • 0 files changed by more than 25 contributors (0 lines of code)
    • 0 files changed by 11-25 contributors (0 lines of code)
    • 1 file changed by 6-10 contributors (108 lines of code)
    • 8 files changed by 2-5 contributors (998 lines of code)
    • 5 files changed by 1 contributor (292 lines of code)
0% | 0% | 7% | 71% | 20%
Legend:
26+
11-25
6-10
2-5
1

explore: grouped by folders | grouped by contributors count | data
File Change Frequency per File Extension
py, rs, md, toml, txt, in, svg, gitignore, json, yaml
File Change Frequency per Extension
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
py0% | 0% | 0% | 64% | 35%
rs0% | 0% | 0% | 71% | 28%
toml0% | 0% | 0% | 100% | 0%
in0% | 0% | 0% | 0% | 100%
File Change Frequency per Logical Decomposition
primary
primary (file change frequency)
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
tiktoken0% | 0% | 0% | 65% | 34%
src0% | 0% | 0% | 71% | 28%
tiktoken_ext0% | 0% | 0% | 100% | 0%
ROOT0% | 0% | 0% | 71% | 28%
scripts0% | 0% | 0% | 0% | 100%
Most Frequently Changed Files (Top 15)

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
36 - 2022-12-15 2025-02-14 20 2 shantanu@openai.com 12621235+hauntsaninja@users...
Cargo.toml
in root
 23 - 2023-02-03 2025-02-14 15 3 shantanu@openai.com 12621235+hauntsaninja@users...
model.py
in tiktoken
 77 2 2023-02-03 2025-02-14 12 5 shantanu@openai.com 12621235+hauntsaninja@users...
core.py
in tiktoken
 205 28 2022-12-15 2025-02-14 12 5 shantanu@openai.com 12621235+hauntsaninja@users...
openai_public.py
in tiktoken_ext
 108 6 2022-12-15 2024-10-03 11 6 shantanu@openai.com pap.lorinc@gmail.com
load.py
in tiktoken
 114 7 2023-01-03 2025-02-14 10 3 shantanu@openai.com 12621235+hauntsaninja@users...
lib.rs
in src
 393 12 2023-02-26 2025-03-09 10 5 nickstathas@outlook.com 12621235+hauntsaninja@users...
__init__.py
in tiktoken
 6 - 2023-02-03 2025-02-14 4 1 shantanu@openai.com shantanu@openai.com
setup.py
in root
 16 - 2022-12-15 2025-02-14 4 2 shantanu@openai.com 12621235+hauntsaninja@users...
_educational.py
in tiktoken
 134 11 2023-06-07 2024-10-02 4 2 shantanu@openai.com shantanu@openai.com
registry.py
in tiktoken
 70 4 2023-01-03 2024-10-02 3 1 shantanu@openai.com shantanu@openai.com
in
 8 - 2022-12-15 2023-01-03 2 1 shantanu@openai.com shantanu@openai.com
redact.py
in scripts
 52 3 2023-01-03 2023-05-07 2 1 shantanu@openai.com shantanu@openai.com
py.rs
in src
 156 11 2025-02-14 2025-03-09 2 1 12621235+hauntsaninja@users... 12621235+hauntsaninja@users...
benchmark.py
in scripts
 28 1
Files With Most Contributors (Top 15)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
openai_public.py
in tiktoken_ext
 108 6 2022-12-15 2024-10-03 11 6 shantanu@openai.com pap.lorinc@gmail.com
core.py
in tiktoken
 205 28 2022-12-15 2025-02-14 12 5 shantanu@openai.com 12621235+hauntsaninja@users...
model.py
in tiktoken
 77 2 2023-02-03 2025-02-14 12 5 shantanu@openai.com 12621235+hauntsaninja@users...
lib.rs
in src
 393 12 2023-02-26 2025-03-09 10 5 nickstathas@outlook.com 12621235+hauntsaninja@users...
Cargo.toml
in root
 23 - 2023-02-03 2025-02-14 15 3 shantanu@openai.com 12621235+hauntsaninja@users...
load.py
in tiktoken
 114 7 2023-01-03 2025-02-14 10 3 shantanu@openai.com 12621235+hauntsaninja@users...
36 - 2022-12-15 2025-02-14 20 2 shantanu@openai.com 12621235+hauntsaninja@users...
_educational.py
in tiktoken
 134 11 2023-06-07 2024-10-02 4 2 shantanu@openai.com shantanu@openai.com
setup.py
in root
 16 - 2022-12-15 2025-02-14 4 2 shantanu@openai.com 12621235+hauntsaninja@users...
__init__.py
in tiktoken
 6 - 2023-02-03 2025-02-14 4 1 shantanu@openai.com shantanu@openai.com
registry.py
in tiktoken
 70 4 2023-01-03 2024-10-02 3 1 shantanu@openai.com shantanu@openai.com
redact.py
in scripts
 52 3 2023-01-03 2023-05-07 2 1 shantanu@openai.com shantanu@openai.com
py.rs
in src
 156 11 2025-02-14 2025-03-09 2 1 12621235+hauntsaninja@users... 12621235+hauntsaninja@users...
in
 8 - 2022-12-15 2023-01-03 2 1 shantanu@openai.com shantanu@openai.com
benchmark.py
in scripts
 28 1
Files With Least Contributors (Top 15)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
benchmark.py
in scripts
 28 1
py.rs
in src
 156 11 2025-02-14 2025-03-09 2 1 12621235+hauntsaninja@users... 12621235+hauntsaninja@users...
registry.py
in tiktoken
 70 4 2023-01-03 2024-10-02 3 1 shantanu@openai.com shantanu@openai.com
redact.py
in scripts
 52 3 2023-01-03 2023-05-07 2 1 shantanu@openai.com shantanu@openai.com
in
 8 - 2022-12-15 2023-01-03 2 1 shantanu@openai.com shantanu@openai.com
__init__.py
in tiktoken
 6 - 2023-02-03 2025-02-14 4 1 shantanu@openai.com shantanu@openai.com
_educational.py
in tiktoken
 134 11 2023-06-07 2024-10-02 4 2 shantanu@openai.com shantanu@openai.com
36 - 2022-12-15 2025-02-14 20 2 shantanu@openai.com 12621235+hauntsaninja@users...
setup.py
in root
 16 - 2022-12-15 2025-02-14 4 2 shantanu@openai.com 12621235+hauntsaninja@users...
load.py
in tiktoken
 114 7 2023-01-03 2025-02-14 10 3 shantanu@openai.com 12621235+hauntsaninja@users...
Cargo.toml
in root
 23 - 2023-02-03 2025-02-14 15 3 shantanu@openai.com 12621235+hauntsaninja@users...
lib.rs
in src
 393 12 2023-02-26 2025-03-09 10 5 nickstathas@outlook.com 12621235+hauntsaninja@users...
core.py
in tiktoken
 205 28 2022-12-15 2025-02-14 12 5 shantanu@openai.com 12621235+hauntsaninja@users...
model.py
in tiktoken
 77 2 2023-02-03 2025-02-14 12 5 shantanu@openai.com 12621235+hauntsaninja@users...
openai_public.py
in tiktoken_ext
 108 6 2022-12-15 2024-10-03 11 6 shantanu@openai.com pap.lorinc@gmail.com
Correlations

File Size vs. Number of Changes: 14 points

src/lib.rs x: 393 lines of code y: 10 # changes src/py.rs x: 156 lines of code y: 2 # changes pyproject.toml x: 36 lines of code y: 20 # changes Cargo.toml x: 23 lines of code y: 15 # changes tiktoken/__init__.py x: 6 lines of code y: 4 # changes tiktoken/core.py x: 205 lines of code y: 12 # changes tiktoken/load.py x: 114 lines of code y: 10 # changes tiktoken/model.py x: 77 lines of code y: 12 # changes setup.py x: 16 lines of code y: 4 # changes tiktoken/_educational.py x: 134 lines of code y: 4 # changes tiktoken/registry.py x: 70 lines of code y: 3 # changes tiktoken_ext/openai_public.py x: 108 lines of code y: 11 # changes scripts/redact.py x: 52 lines of code y: 2 # changes MANIFEST.in x: 8 lines of code y: 2 # changes
20.0
# changes
  min: 2.0
  average: 7.93
  25th percentile: 2.75
  median: 7.0
  75th percentile: 12.0
  max: 20.0
0 393.0
lines of code
min: 6.0 | average: 99.86 | 25th percentile: 21.25 | median: 73.5 | 75th percentile: 139.5 | max: 393.0

Number of Contributors vs. Number of Changes: 14 points

src/lib.rs x: 5 # contributors y: 10 # changes src/py.rs x: 1 # contributors y: 2 # changes pyproject.toml x: 2 # contributors y: 20 # changes Cargo.toml x: 3 # contributors y: 15 # changes tiktoken/__init__.py x: 1 # contributors y: 4 # changes tiktoken/core.py x: 5 # contributors y: 12 # changes tiktoken/load.py x: 3 # contributors y: 10 # changes setup.py x: 2 # contributors y: 4 # changes tiktoken/registry.py x: 1 # contributors y: 3 # changes tiktoken_ext/openai_public.py x: 6 # contributors y: 11 # changes
20.0
# changes
  min: 2.0
  average: 7.93
  25th percentile: 2.75
  median: 7.0
  75th percentile: 12.0
  max: 20.0
0 6.0
# contributors
min: 1.0 | average: 2.71 | 25th percentile: 1.0 | median: 2.0 | 75th percentile: 5.0 | max: 6.0

Number of Contributors vs. File Size: 14 points

src/lib.rs x: 5 # contributors y: 393 lines of code src/py.rs x: 1 # contributors y: 156 lines of code pyproject.toml x: 2 # contributors y: 36 lines of code Cargo.toml x: 3 # contributors y: 23 lines of code tiktoken/__init__.py x: 1 # contributors y: 6 lines of code tiktoken/core.py x: 5 # contributors y: 205 lines of code tiktoken/load.py x: 3 # contributors y: 114 lines of code tiktoken/model.py x: 5 # contributors y: 77 lines of code setup.py x: 2 # contributors y: 16 lines of code tiktoken/_educational.py x: 2 # contributors y: 134 lines of code tiktoken/registry.py x: 1 # contributors y: 70 lines of code tiktoken_ext/openai_public.py x: 6 # contributors y: 108 lines of code scripts/redact.py x: 1 # contributors y: 52 lines of code MANIFEST.in x: 1 # contributors y: 8 lines of code
393.0
lines of code
  min: 6.0
  average: 99.86
  25th percentile: 21.25
  median: 73.5
  75th percentile: 139.5
  max: 393.0
0 6.0
# contributors
min: 1.0 | average: 2.71 | 25th percentile: 1.0 | median: 2.0 | 75th percentile: 5.0 | max: 6.0