apache / beam - Source Code Overview

Analysis scope, overview of main, test, generated, deployment, build, and other code.

Source Code Analysis Scope

Files includes and excluded from analyses

txt

properties

proto

sass

pxd

mod

hcl

ini

tpl

gitattributes

cfg

prettierignore

pyi

htaccess

gitmodules

eslintignore

58 extensions are included in analyses: java, py, yaml, go, md, dart, txt, gradle, json, svg, ts, sh, tf, kt, sql, scss, html, ipynb, groovy, js, properties, xml, proto, tsx, sass, pxd, kts, mod, css, tfvars, avsc, pyx, bat, hcl, ini, toml, p8, gitignore, tpl, pb, ps1, gitattributes, thrift, cc, cfg, dockerfile, h, prettierignore, pyi, htaccess, in, g4, gitmodules, eslintignore, rs, scala, ftl, rdf
9 criteria are used to exclude files from analysis:

exclude files with path like ".*/cache/.*" (Caches) (22 files).
exclude files with path like ".*/gradle/wrapper/.*" (Gradle) (4 files).
exclude files with path like ".*/[.][a-zA-Z0-9_]+.*" (Hidden files and folders) (648 files).
exclude files with path like ".*/testdata/.*" (Test data) (2 files).
exclude files with path like ".*/docs/.*" (Documentation) (2 files).
exclude files with path like ".*/[Jj]avadoc/.*" (Generated documentation) (2 files).
exclude files with path like ".*/extern(al)?/.*" (Dependencies) (15 files).
exclude files with path like ".*/[Vv]endors?/.*" (Dependencies) (4 files).
exclude files with path like ".*/bootstrap[.](js|css|less|scss|styl)" (Bootstrap) (1 file).

The source code folder contains 14,833 files:

12,896 files are included in analyses.
1,937 files are excluded from analyses:

Explore all files....

Overview of Analyzed Files

Basic stats on analyzed files

Intro

For analysis purposes we separate files in scope into several categories: main, test, generated, deployment and build, and other.

The main category contains all manually created source code files that are being used in the production.
Files in the main category are used as input for other analyses: logical decomposition, concerns, duplication, file size, unit size, and conditional complexity.
Test source code files are used only for testing of the product. These files are normally not deployed to production.
Build and deployment source code files are used to configure or support build and deployment process.
Generated source code files are automatically generated files that have not been manually changed after generation.
While a source code folder may contain a number of files, we are primarily interested in the source code files that are being written and maintained by developers.
Files containing binaries, documentation, or third-party libraries, for instance, are excluded from analysis. The exception are third-party libraries that have been changed by developers.

Main Code

All manually created or maintained source code that defines logic of the product that is run in a production environment.

proto

pxd

sass

pyi

cfg

Explore: circles | sunburst

The following criteria are used to filter files:

files with paths like ".*".

7108 files match defined criteria (774,010 LOC, 100.0% vs. main code):

4,139 *.java files (447,766 LOC)
674 *.py files (114,349 LOC)
625 *.go files (107,240 LOC)
129 *.ts files (27,486 LOC)
401 *.dart files (20,328 LOC)
589 *.yaml files (15,523 LOC)
98 *.scss files (9,510 LOC)
17 *.proto files (6,166 LOC)
103 *.sql files (4,461 LOC)
75 *.html files (4,176 LOC)
23 *.groovy files (3,869 LOC)
28 *.js files (2,600 LOC)
82 *.tf files (2,328 LOC)
13 *.kts files (2,204 LOC)
43 *.kt files (1,225 LOC)
10 *.tsx files (869 LOC)
14 *.pxd files (792 LOC)
14 *.sass files (706 LOC)
5 *.pyx files (548 LOC)
8 *.xml files (468 LOC)
1 *.pyi files (376 LOC)
1 *.ftl files (355 LOC)
6 *.css files (216 LOC)
1 *.ipynb files (172 LOC)
2 *.toml files (128 LOC)
2 *.h files (49 LOC)
2 *.cfg files (41 LOC)
1 *.rdf files (23 LOC)
1 *.in files (22 LOC)
1 *.scala files (14 LOC)

" *.java" is biggest, containing 57.85% of LOC.
" *.scala" is smallest, containing 0% of LOC.

Test Code

Used only for testing of the product. Normally not deployed in a production environment.

proto

Explore: circles | sunburst

The following criteria are used to filter files:

files with paths like ".*_test[.].*".
files with paths like ".*/test_.*".
files with paths like ".*/[Tt]est/.*".
files with paths like ".*_mock[.][a-zA-Z0-9_\-]+".
files with paths like ".*/[Tt]ests/.*".
files with paths like ".*[-]tests[-].*".
files with paths like ".*[-]test[-].*".
files with paths like ".*[-]tests/.*".
files with paths like ".*/test[.].*".
files with paths like ".*/tests_.*".
files with paths like ".*_tests[.].*".
files with paths like ".*/jest[.][a-zA-Z0-9\.]+".
files with paths like ".*[.][Tt]est[.].*".
files with paths like ".*[.]test[.].*".
files with paths like ".*__tests__.*".
files with paths like ".*/mock[a-zA-Z0-9_\- ]+/.*".
files with paths like ".*/testing[.].*".

3255 files match defined criteria (596,198 LOC, 77.0% vs. main code):

2,188 *.java files (420,095 LOC)
440 *.py files (96,369 LOC)
404 *.go files (63,895 LOC)
74 *.dart files (6,488 LOC)
36 *.yaml files (2,268 LOC)
14 *.ts files (2,154 LOC)
39 *.kt files (980 LOC)
7 *.sh files (797 LOC)
10 *.proto files (704 LOC)
8 *.tsx files (678 LOC)
11 *.gradle files (516 LOC)
8 *.xml files (459 LOC)
2 *.ipynb files (284 LOC)
6 *.avsc files (239 LOC)
1 *.sql files (121 LOC)
3 *.p8 files (54 LOC)
2 *.thrift files (45 LOC)
1 *.dockerfile files (38 LOC)
1 *.js files (14 LOC)

" *.java" is biggest, containing 70.46% of LOC.
" *.js" is smallest, containing 0% of LOC.

Generated Code

Automatically generated files, not manually changed after generation.

Explore: circles | sunburst

The following criteria are used to filter files:

files with paths like ".*[.]dart" AND any line of content like ".*GENERATED CODE.*DO NOT MODIFY.*".
files with paths like ".*/package[-]lock[.]json".
files with paths like ".*/generated/.*".

27 files match defined criteria (11,521 LOC, 1.5% vs. main code):

3 *.json files (10,485 LOC)
22 *.dart files (991 LOC)
1 *.g4 files (41 LOC)
1 *.java files (4 LOC)

" *.json" is biggest, containing 91.01% of LOC.
" *.java" is smallest, containing 0.03% of LOC.

Build and Deployment Code

Source code used to configure or support build and deployment process.

Explore: circles | sunburst

The following criteria are used to filter files:

files with paths like ".*[.]gradle".
files with paths like ".*[.]sh".
files with paths like ".*/package[-]lock[.]json".
files with paths like ".*/package[.]json".
files with paths like ".*[.]git[a-z]+".
files with paths like ".*/[.]gitignore".
files with paths like ".*[.]bat".
files with paths like ".*/[.]gitmodules".
files with paths like ".*/pom[.]xml".
files with paths like ".*[.]dockerfile".
files with paths like ".*/assembly[.]xml".
files with paths like ".*/jenkins/.*[.]groovy".
files with paths like ".*/[.]gitattributes".

330 files match defined criteria (17,946 LOC, 2.3% vs. main code):

249 *.gradle files (12,544 LOC)
71 *.sh files (4,394 LOC)
5 *.xml files (685 LOC)
4 *.bat files (295 LOC)
1 *.dockerfile files (28 LOC)

" *.gradle" is biggest, containing 69.9% of LOC.
" *.dockerfile" is smallest, containing 0.16% of LOC.

Other Code

txt

properties

ini

mod

Explore: circles | sunburst

The following criteria are used to filter files:

files with paths like ".*[.]md".
files with paths like ".*/README[.][a-z0-9]+".
files with paths like ".*/[Ee]xamples/.*".
files with paths like ".*[.]svg".
files with paths like ".*/go[.]mod".
files with paths like ".*[.]txt".
files with paths like ".*[.]properties".
files with paths like ".*[.]json".
files with paths like ".*/[Dd]ocumentation/.*".
files with paths like ".*/[.]gitignore".
files with paths like ".*/[Ss]amples/.*".
files with paths like ".*/[.]prettierignore".
files with paths like ".*/checkstyle[.]xml".
files with paths like ".*/checkstyle.*".
files with paths like ".*[.]pb".
files with paths like ".*/[.]eslintignore".
files with paths like ".*[.]ini".
files with paths like ".*/vendor/.*".
files with paths like ".*/[.]htaccess".

2179 files match defined criteria (225,565 LOC, 29.1% vs. main code):

986 *.md files (83,449 LOC)
110 *.json files (34,640 LOC)
43 *.ipynb files (28,375 LOC)
361 *.py files (18,669 LOC)
264 *.java files (17,371 LOC)
58 *.txt files (17,205 LOC)
116 *.svg files (12,383 LOC)
53 *.go files (4,347 LOC)
24 *.kt files (1,914 LOC)
32 *.dart files (1,325 LOC)
30 *.properties files (1,055 LOC)
56 *.yaml files (1,033 LOC)
14 *.gradle files (952 LOC)
5 *.xml files (868 LOC)
3 *.ini files (637 LOC)
3 *.pb files (384 LOC)
6 *.mod files (369 LOC)
7 *.html files (340 LOC)
3 *.ts files (77 LOC)
1 *.rs files (64 LOC)
1 *.sh files (40 LOC)
1 *.toml files (36 LOC)
2 *.cc files (32 LOC)

" *.md" is biggest, containing 37% of LOC.
" *.cc" is smallest, containing 0.01% of LOC.

Analyzers

Info about analyzers used for source code examinations.

*.java files are analyzed with JavaAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Advanced heuristic dependency analysis (based on package names)

*.py files are analyzed with PythonAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis

*.go files are analyzed with GoLangAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis

*.ts files are analyzed with TypeScriptAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

No dependency analysis

*.dart files are analyzed with CppAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Advanced heuristic dependency analysis

*.yaml files are analyzed with YamlAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.scss files are analyzed with ScssAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.proto files are analyzed with DefaultLanguageAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Basic code cleaning (empty lines removed for LOC calculations and duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.sql files are analyzed with SqlAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.html files are analyzed with HtmlAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Advanced heuristic dependency analysis

*.groovy files are analyzed with GroovyAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis

*.js files are analyzed with JavaScriptAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

No dependency analysis

*.tf files are analyzed with YamlAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.kts files are analyzed with KotlinAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis (based on imports and package statements)

*.kt files are analyzed with KotlinAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis (based on imports and package statements)

*.tsx files are analyzed with TypeScriptAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

No dependency analysis

*.pxd files are analyzed with PythonAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis

*.sass files are analyzed with SassAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.pyx files are analyzed with PythonAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis

*.xml files are analyzed with XmlAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.pyi files are analyzed with PythonAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis

*.ftl files are analyzed with DefaultLanguageAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Basic code cleaning (empty lines removed for LOC calculations and duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.css files are analyzed with CssAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.ipynb files are analyzed with DefaultLanguageAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Basic code cleaning (empty lines removed for LOC calculations and duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.toml files are analyzed with DefaultLanguageAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Basic code cleaning (empty lines removed for LOC calculations and duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.h files are analyzed with CppAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Advanced heuristic dependency analysis

*.cfg files are analyzed with CfgAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.rdf files are analyzed with XmlAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

No unit size analysis

No conditional complexity analysis

No dependency analysis

*.in files are analyzed with RustAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

No dependency analysis

*.scala files are analyzed with ScalaAnalyzer:

All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)

Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)

Unit size analysis

Conditional complexity analysis

Basic heuristic dependency analysis

2025-05-07 16:01