openai/tiktoken
File Churn
File change frequency (churn) shows the distribution of file updates (days with at least one commit).
Overview
File Change Frequency Overall
  • There are 12 files with 1,594 lines of code.
    • 0 files changed more than 100 times (0 lines of code)
    • 0 files changed 51-100 times (0 lines of code)
    • 0 files changed 21-50 times (0 lines of code)
    • 6 files changed 6-20 times (1,079 lines of code)
    • 6 files changed 1-5 times (515 lines of code)
0% | 0% | 0% | 67% | 32%
Legend:
101+
51-100
21-50
6-20
1-5

explore: grouped by folders | grouped by update frequency | data
Contributors Count Frequency Overall
  • There are 12 files with 1,594 lines of code.
    • 0 files changed by more than 25 contributors (0 lines of code)
    • 0 files changed by 11-25 contributors (0 lines of code)
    • 3 files changed by 6-10 contributors (740 lines of code)
    • 5 files changed by 2-5 contributors (683 lines of code)
    • 4 files changed by 1 contributor (171 lines of code)
0% | 0% | 46% | 42% | 10%
Legend:
26+
11-25
6-10
2-5
1

explore: grouped by folders | grouped by contributors count | data
File Change Frequency per File Extension
py, txt, rs, md, toml, in, svg, gitignore, yml
File Change Frequency per Extension
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
py0% | 0% | 0% | 65% | 34%
rs0% | 0% | 0% | 71% | 28%
in0% | 0% | 0% | 0% | 100%
File Change Frequency per Logical Decomposition
primary
primary (file change frequency)
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
src0% | 0% | 0% | 71% | 28%
tiktoken0% | 0% | 0% | 67% | 32%
tiktoken_ext0% | 0% | 0% | 100% | 0%
scripts0% | 0% | 0% | 0% | 100%
ROOT0% | 0% | 0% | 0% | 100%
Most Frequently Changed Files (Top 13)

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
model.py
in tiktoken
 81 2 2023-02-03 2026-08-17 16 6 +151 / -37 shantanu@openai.com 12621235+hauntsaninja@users...
core.py
in tiktoken
 211 28 2022-12-15 2026-05-15 15 5 +171 / -32 shantanu@openai.com 12621235+hauntsaninja@users...
load.py
in tiktoken
 122 7 2023-01-03 2026-02-08 13 3 +131 / -57 shantanu@openai.com 12621235+hauntsaninja@users...
lib.rs
in src
 521 16 2023-02-26 2026-02-08 13 6 +605 / -462 nickstathas@outlook.com 12621235+hauntsaninja@users...
openai_public.py
in tiktoken_ext
 138 7 2022-12-15 2025-08-05 12 6 +156 / -35 shantanu@openai.com shantanu@openai.com
__init__.py
in tiktoken
 6 - 2023-02-03 2026-08-17 9 2 +11 / -6 shantanu@openai.com 12621235+hauntsaninja@users...
py.rs
in src
 210 12 2025-02-14 2025-09-30 5 3 +312 / -57 12621235+hauntsaninja@users... 12621235+hauntsaninja@users...
_educational.py
in tiktoken
 134 11 2023-06-08 2024-10-02 4 2 +236 / -13 shantanu@openai.com shantanu@openai.com
registry.py
in tiktoken
 70 4 2023-01-03 2024-10-02 3 1 +39 / -14 shantanu@openai.com shantanu@openai.com
in
 8 - 2022-12-15 2023-01-03 2 1 +3 / -0 shantanu@openai.com shantanu@openai.com
redact.py
in scripts
 52 3 2023-01-03 2023-05-07 2 1 +3 / -1 shantanu@openai.com shantanu@openai.com
41 1 2025-08-08 2025-08-08 1 1 +56 / -0 shantanu@openai.com shantanu@openai.com
benchmark.py
in scripts
 28 1
Files With Most Churn (Top 13)
Based on the number of lines added and deleted across all commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
lib.rs
in src
 521 16 2023-02-26 2026-02-08 13 6 +605 / -462 nickstathas@outlook.com 12621235+hauntsaninja@users...
py.rs
in src
 210 12 2025-02-14 2025-09-30 5 3 +312 / -57 12621235+hauntsaninja@users... 12621235+hauntsaninja@users...
_educational.py
in tiktoken
 134 11 2023-06-08 2024-10-02 4 2 +236 / -13 shantanu@openai.com shantanu@openai.com
core.py
in tiktoken
 211 28 2022-12-15 2026-05-15 15 5 +171 / -32 shantanu@openai.com 12621235+hauntsaninja@users...
openai_public.py
in tiktoken_ext
 138 7 2022-12-15 2025-08-05 12 6 +156 / -35 shantanu@openai.com shantanu@openai.com
model.py
in tiktoken
 81 2 2023-02-03 2026-08-17 16 6 +151 / -37 shantanu@openai.com 12621235+hauntsaninja@users...
load.py
in tiktoken
 122 7 2023-01-03 2026-02-08 13 3 +131 / -57 shantanu@openai.com 12621235+hauntsaninja@users...
41 1 2025-08-08 2025-08-08 1 1 +56 / -0 shantanu@openai.com shantanu@openai.com
registry.py
in tiktoken
 70 4 2023-01-03 2024-10-02 3 1 +39 / -14 shantanu@openai.com shantanu@openai.com
__init__.py
in tiktoken
 6 - 2023-02-03 2026-08-17 9 2 +11 / -6 shantanu@openai.com 12621235+hauntsaninja@users...
redact.py
in scripts
 52 3 2023-01-03 2023-05-07 2 1 +3 / -1 shantanu@openai.com shantanu@openai.com
in
 8 - 2022-12-15 2023-01-03 2 1 +3 / -0 shantanu@openai.com shantanu@openai.com
benchmark.py
in scripts
 28 1
Files With Most Contributors (Top 13)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
model.py
in tiktoken
 81 2 2023-02-03 2026-08-17 16 6 +151 / -37 shantanu@openai.com 12621235+hauntsaninja@users...
lib.rs
in src
 521 16 2023-02-26 2026-02-08 13 6 +605 / -462 nickstathas@outlook.com 12621235+hauntsaninja@users...
openai_public.py
in tiktoken_ext
 138 7 2022-12-15 2025-08-05 12 6 +156 / -35 shantanu@openai.com shantanu@openai.com
core.py
in tiktoken
 211 28 2022-12-15 2026-05-15 15 5 +171 / -32 shantanu@openai.com 12621235+hauntsaninja@users...
load.py
in tiktoken
 122 7 2023-01-03 2026-02-08 13 3 +131 / -57 shantanu@openai.com 12621235+hauntsaninja@users...
py.rs
in src
 210 12 2025-02-14 2025-09-30 5 3 +312 / -57 12621235+hauntsaninja@users... 12621235+hauntsaninja@users...
__init__.py
in tiktoken
 6 - 2023-02-03 2026-08-17 9 2 +11 / -6 shantanu@openai.com 12621235+hauntsaninja@users...
_educational.py
in tiktoken
 134 11 2023-06-08 2024-10-02 4 2 +236 / -13 shantanu@openai.com shantanu@openai.com
registry.py
in tiktoken
 70 4 2023-01-03 2024-10-02 3 1 +39 / -14 shantanu@openai.com shantanu@openai.com
in
 8 - 2022-12-15 2023-01-03 2 1 +3 / -0 shantanu@openai.com shantanu@openai.com
redact.py
in scripts
 52 3 2023-01-03 2023-05-07 2 1 +3 / -1 shantanu@openai.com shantanu@openai.com
41 1 2025-08-08 2025-08-08 1 1 +56 / -0 shantanu@openai.com shantanu@openai.com
benchmark.py
in scripts
 28 1
Files With Least Contributors (Top 13)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
benchmark.py
in scripts
 28 1
registry.py
in tiktoken
 70 4 2023-01-03 2024-10-02 3 1 +39 / -14 shantanu@openai.com shantanu@openai.com
redact.py
in scripts
 52 3 2023-01-03 2023-05-07 2 1 +3 / -1 shantanu@openai.com shantanu@openai.com
41 1 2025-08-08 2025-08-08 1 1 +56 / -0 shantanu@openai.com shantanu@openai.com
in
 8 - 2022-12-15 2023-01-03 2 1 +3 / -0 shantanu@openai.com shantanu@openai.com
_educational.py
in tiktoken
 134 11 2023-06-08 2024-10-02 4 2 +236 / -13 shantanu@openai.com shantanu@openai.com
__init__.py
in tiktoken
 6 - 2023-02-03 2026-08-17 9 2 +11 / -6 shantanu@openai.com 12621235+hauntsaninja@users...
py.rs
in src
 210 12 2025-02-14 2025-09-30 5 3 +312 / -57 12621235+hauntsaninja@users... 12621235+hauntsaninja@users...
load.py
in tiktoken
 122 7 2023-01-03 2026-02-08 13 3 +131 / -57 shantanu@openai.com 12621235+hauntsaninja@users...
core.py
in tiktoken
 211 28 2022-12-15 2026-05-15 15 5 +171 / -32 shantanu@openai.com 12621235+hauntsaninja@users...
lib.rs
in src
 521 16 2023-02-26 2026-02-08 13 6 +605 / -462 nickstathas@outlook.com 12621235+hauntsaninja@users...
openai_public.py
in tiktoken_ext
 138 7 2022-12-15 2025-08-05 12 6 +156 / -35 shantanu@openai.com shantanu@openai.com
model.py
in tiktoken
 81 2 2023-02-03 2026-08-17 16 6 +151 / -37 shantanu@openai.com 12621235+hauntsaninja@users...
Correlations

File Size vs. Number of Changes: 12 points

tiktoken/__init__.py x: 6 lines of code y: 9 # changes tiktoken/model.py x: 81 lines of code y: 16 # changes tiktoken/core.py x: 211 lines of code y: 15 # changes src/lib.rs x: 521 lines of code y: 13 # changes tiktoken/load.py x: 122 lines of code y: 13 # changes src/py.rs x: 210 lines of code y: 5 # changes scripts/wheel_download.py x: 41 lines of code y: 1 # changes tiktoken_ext/openai_public.py x: 138 lines of code y: 12 # changes tiktoken/_educational.py x: 134 lines of code y: 4 # changes tiktoken/registry.py x: 70 lines of code y: 3 # changes scripts/redact.py x: 52 lines of code y: 2 # changes MANIFEST.in x: 8 lines of code y: 2 # changes
16.0
# changes
  min: 1.0
  average: 7.92
  25th percentile: 2.25
  median: 7.0
  75th percentile: 13.0
  max: 16.0
0 521.0
lines of code
min: 6.0 | average: 132.83 | 25th percentile: 43.75 | median: 101.5 | 75th percentile: 192.0 | max: 521.0

Number of Contributors vs. Number of Changes: 12 points

tiktoken/__init__.py x: 2 # contributors y: 9 # changes tiktoken/model.py x: 6 # contributors y: 16 # changes tiktoken/core.py x: 5 # contributors y: 15 # changes src/lib.rs x: 6 # contributors y: 13 # changes tiktoken/load.py x: 3 # contributors y: 13 # changes src/py.rs x: 3 # contributors y: 5 # changes scripts/wheel_download.py x: 1 # contributors y: 1 # changes tiktoken_ext/openai_public.py x: 6 # contributors y: 12 # changes tiktoken/_educational.py x: 2 # contributors y: 4 # changes tiktoken/registry.py x: 1 # contributors y: 3 # changes scripts/redact.py x: 1 # contributors y: 2 # changes
16.0
# changes
  min: 1.0
  average: 7.92
  25th percentile: 2.25
  median: 7.0
  75th percentile: 13.0
  max: 16.0
0 6.0
# contributors
min: 1.0 | average: 3.08 | 25th percentile: 1.0 | median: 2.5 | 75th percentile: 5.75 | max: 6.0

Number of Contributors vs. File Size: 12 points

tiktoken/__init__.py x: 2 # contributors y: 6 lines of code tiktoken/model.py x: 6 # contributors y: 81 lines of code tiktoken/core.py x: 5 # contributors y: 211 lines of code src/lib.rs x: 6 # contributors y: 521 lines of code tiktoken/load.py x: 3 # contributors y: 122 lines of code src/py.rs x: 3 # contributors y: 210 lines of code scripts/wheel_download.py x: 1 # contributors y: 41 lines of code tiktoken_ext/openai_public.py x: 6 # contributors y: 138 lines of code tiktoken/_educational.py x: 2 # contributors y: 134 lines of code tiktoken/registry.py x: 1 # contributors y: 70 lines of code scripts/redact.py x: 1 # contributors y: 52 lines of code MANIFEST.in x: 1 # contributors y: 8 lines of code
521.0
lines of code
  min: 6.0
  average: 132.83
  25th percentile: 43.75
  median: 101.5
  75th percentile: 192.0
  max: 521.0
0 6.0
# contributors
min: 1.0 | average: 3.08 | 25th percentile: 1.0 | median: 2.5 | 75th percentile: 5.75 | max: 6.0

Number of Changes vs. Line Churn: 27 points

CHANGELOG.md x: 18 # changes y: 141 line churn Cargo.toml x: 23 # changes y: 101 line churn pyproject.toml x: 27 # changes y: 112 line churn tiktoken/__init__.py x: 9 # changes y: 17 line churn tiktoken/model.py x: 16 # changes y: 188 line churn .github/workflows/build_wheels.yml x: 16 # changes y: 232 line churn tiktoken/core.py x: 15 # changes y: 203 line churn src/lib.rs x: 13 # changes y: 1067 line churn tests/test_encoding.py x: 6 # changes y: 312 line churn tiktoken/load.py x: 13 # changes y: 188 line churn src/py.rs x: 5 # changes y: 369 line churn scripts/wheel_download.py x: 1 # changes y: 56 line churn tests/test_misc.py x: 3 # changes y: 30 line churn tiktoken_ext/openai_public.py x: 12 # changes y: 191 line churn README.md x: 12 # changes y: 135 line churn setup.py x: 4 # changes y: 12 line churn tests/test_pickle.py x: 1 # changes y: 23 line churn tiktoken/_educational.py x: 4 # changes y: 249 line churn tiktoken/registry.py x: 3 # changes y: 53 line churn .gitignore x: 1 # changes y: 1 line churn tests/test_offsets.py x: 1 # changes y: 79 line churn scripts/redact.py x: 2 # changes y: 4 line churn tests/test_simple_public.py x: 6 # changes y: 42 line churn Makefile x: 1 # changes y: 49 line churn
1067.0
line churn
  min: 1.0
  average: 143.7
  25th percentile: 22.0
  median: 79.0
  75th percentile: 191.0
  max: 1067.0
0 27.0
# changes
min: 1.0 | average: 8.0 | 25th percentile: 1.0 | median: 5.0 | 75th percentile: 13.0 | max: 27.0

Number of Contributors vs. Line Churn: 12 points

tiktoken/__init__.py x: 2 # contributors y: 17 line churn tiktoken/model.py x: 6 # contributors y: 188 line churn tiktoken/core.py x: 5 # contributors y: 203 line churn src/lib.rs x: 6 # contributors y: 1067 line churn tiktoken/load.py x: 3 # contributors y: 188 line churn src/py.rs x: 3 # contributors y: 369 line churn scripts/wheel_download.py x: 1 # contributors y: 56 line churn tiktoken_ext/openai_public.py x: 6 # contributors y: 191 line churn tiktoken/_educational.py x: 2 # contributors y: 249 line churn tiktoken/registry.py x: 1 # contributors y: 53 line churn scripts/redact.py x: 1 # contributors y: 4 line churn
1067.0
line churn
  min: 3.0
  average: 215.67
  25th percentile: 26.0
  median: 188.0
  75th percentile: 237.5
  max: 1067.0
0 6.0
# contributors
min: 1.0 | average: 3.08 | 25th percentile: 1.0 | median: 2.5 | 75th percentile: 5.75 | max: 6.0

File Size vs. Line Churn: 12 points

tiktoken/__init__.py x: 6 lines of code y: 17 line churn tiktoken/model.py x: 81 lines of code y: 188 line churn tiktoken/core.py x: 211 lines of code y: 203 line churn src/lib.rs x: 521 lines of code y: 1067 line churn tiktoken/load.py x: 122 lines of code y: 188 line churn src/py.rs x: 210 lines of code y: 369 line churn scripts/wheel_download.py x: 41 lines of code y: 56 line churn tiktoken_ext/openai_public.py x: 138 lines of code y: 191 line churn tiktoken/_educational.py x: 134 lines of code y: 249 line churn tiktoken/registry.py x: 70 lines of code y: 53 line churn scripts/redact.py x: 52 lines of code y: 4 line churn MANIFEST.in x: 8 lines of code y: 3 line churn
1067.0
line churn
  min: 3.0
  average: 215.67
  25th percentile: 26.0
  median: 188.0
  75th percentile: 237.5
  max: 1067.0
0 521.0
lines of code
min: 6.0 | average: 132.83 | 25th percentile: 43.75 | median: 101.5 | 75th percentile: 192.0 | max: 521.0