Netflix/spark
Code Volume
Analysis scope, overview of main, test, generated, deployment, build, and other code.
Source Code Analysis Scope
Files includes and excluded from analyses
q
properties
cmd
orc
in
cfg
rmd
gitattributes
ini

scoping details...
  • 35 extensions are included in analyses: scala, q, java, sql, txt, py, md, r, json, xml, sh, js, properties, css, cmd, html, rst, gitignore, yaml, orc, bat, rb, avsc, yml, thrift, sbt, in, c, cfg, rmd, ps1, g4, gitattributes, ini, bash
  • 14 criteria are used to exclude files from analysis:
    • exclude files with path like ".*[.]min[.]js" (Minimized JS library) (2 files).
    • exclude files with path like "(?i).*/jquery.*[.]js" (jQuery files) (1 file).
    • exclude files with path like ".*jquery[.].*[.]js" (jQuery library) (0 files).
    • exclude files with path like ".*[.]bootstrap[.](js|css|less|scss|styl)" (Bootstrap) (1 file).
    • exclude files with path like ".*/bin/.*" (Binaries for distribution) (18 files).
    • exclude files with path like ".*/docs/.*" (Documentation) (119 files).
    • exclude files with path like ".*/docs?/_?(build|themes?|templates?|static)/.*" (Sphinx) (0 files).
    • exclude files with path like ".*/bootstrap[.](js|css|less|scss|styl)" (Bootstrap) (0 files).
    • exclude files with path like ".*/[Vv]endors?/.*" (Dependencies) (0 files).
    • exclude files with path like ".*/extern(al)?/.*" (Dependencies) (174 files).
    • exclude files with path like ".*/_sokrates/.*" (Sokrates files) (0 files).
    • exclude files with path like ".*/_sokrates_landscape/.*" (Sokrates landscape files) (0 files).
    • exclude files with path like ".*/git[-][a-zA-Z0-9_]+[.]txt" (Git data exports for sokrates analyses) (0 files).
    • exclude files with path like ".*/sokrates_.*?[.]json" (Sokrates conventions and configurations) (0 files).
Overview of Analyzed Files
Basic stats on analyzed files
Intro
For analysis purposes we separate files in scope into several categories: main, test, generated, deployment and build, and other.
main315530 LOC (43%) 2,367 files
test327349 LOC (44%) 3,308 files
generated31574 LOC (4%) 82 files
build and deployment8119 LOC (1%) 77 files
other50308 LOC (6%) 801 files
Main Code
All manually created or maintained source code that defines logic of the product that is run in a production environment.
in
Explore:   circles  |  sunburst

scala236598 LOC (74%) 1,707 files
java37873 LOC (12%) 520 files
py34784 LOC (11%) 107 files
js2663 LOC (<1%) 16 files
g41060 LOC (<1%) 1 file
html932 LOC (<1%) 4 files
css575 LOC (<1%) 5 files
thrift518 LOC (<1%) 1 file
xml235 LOC (<1%) 1 file
bash148 LOC (<1%) 1 file
ps198 LOC (<1%) 1 file
in23 LOC (<1%) 1 file
c19 LOC (<1%) 1 file
yaml4 LOC (<1%) 1 file

scope details...
  • The following criteria are used to filter files:
    • files with paths like ".*".
  • 2367 files match defined criteria (315,530 LOC, 100.0% vs. main code):
    • 1,707 scala files (236,598 LOC)
    • 520 java files (37,873 LOC)
    • 107 py files (34,784 LOC)
    • 16 js files (2,663 LOC)
    • 1 g4 files (1,060 LOC)
    • 4 html files (932 LOC)
    • 5 css files (575 LOC)
    • 1 thrift files (518 LOC)
    • 1 xml files (235 LOC)
    • 1 bash files (148 LOC)
    • 1 ps1 files (98 LOC)
    • 1 in files (23 LOC)
    • 1 c files (19 LOC)
    • 1 yaml files (4 LOC)
Test Code
Used only for testing of the product. Normally not deployed in a production environment.
q
orc
Explore:   circles  |  sunburst

scala242399 LOC (74%) 1,196 files
q41429 LOC (12%) 1,559 files
java24997 LOC (7%) 218 files
sql17397 LOC (5%) 295 files
py425 LOC (<1%) 21 files
xml405 LOC (<1%) 7 files
sh214 LOC (<1%) 7 files
thrift35 LOC (<1%) 1 file
yaml35 LOC (<1%) 1 file
orc12 LOC (<1%) 2 files
gitignore1 LOC (<1%) 1 file

scope details...
  • The following criteria are used to filter files:
    • files with paths like ".*/[Tt]est/.*".
    • files with paths like ".*/[Tt]ests/.*".
    • files with paths like ".*[-]tests/.*".
    • files with paths like ".*[-]test[-].*".
    • files with paths like ".*/test_.*".
    • files with paths like ".*/test[.].*".
    • files with paths like ".*[-]tests[-].*".
    • files with paths like ".*/mock[^/]*/.*".
    • files with paths like ".*/[Mm]ocks/.*".
    • files with paths like ".*_test[.].*".
    • files with paths like ".*/test[-]data/.*".
    • files with paths like ".*_tests[.].*".
  • 3308 files match defined criteria (327,349 LOC, 103.7% vs. main code):
    • 1,196 scala files (242,399 LOC)
    • 1,559 q files (41,429 LOC)
    • 218 java files (24,997 LOC)
    • 295 sql files (17,397 LOC)
    • 21 py files (425 LOC)
    • 7 xml files (405 LOC)
    • 7 sh files (214 LOC)
    • 1 thrift files (35 LOC)
    • 1 yaml files (35 LOC)
    • 2 orc files (12 LOC)
    • 1 gitignore files (1 LOC)
Generated Code
Automatically generated files, not manually changed after generation.
Explore:   circles  |  sunburst

java31574 LOC (100%) 82 files

scope details...
  • The following criteria are used to filter files:
    • files with paths like ".*/src/gen/.*".
  • 82 files match defined criteria (31,574 LOC, 10.0% vs. main code). All matches are in java files.
Build and Deployment Code
Source code used to configure or support build and deployment process.
cfg
gitattributes
Explore:   circles  |  sunburst

xml5992 LOC (73%) 28 files
sh1764 LOC (21%) 39 files
py170 LOC (2%) 1 file
gitignore110 LOC (1%) 4 files
yaml35 LOC (<1%) 1 file
bat26 LOC (<1%) 1 file
sbt16 LOC (<1%) 1 file
cfg4 LOC (<1%) 1 file
gitattributes2 LOC (<1%) 1 file

scope details...
  • The following criteria are used to filter files:
    • files with paths like ".*/pom[.]xml".
    • files with paths like ".*[.]sh".
    • files with paths like ".*[.]bat".
    • files with paths like ".*/setup[.]py".
    • files with paths like ".*[.]git[a-z]+".
    • files with paths like ".*/[.]gitignore".
    • files with paths like ".*/setup[.]cfg".
    • files with paths like ".*/assembly[.]xml".
    • files with paths like ".*[.]sbt".
    • files with paths like ".*/appveyor[.]yml".
    • files with paths like ".*/[.]gitattributes".
    • files with paths like ".*/requirements[a-zA-Z0-9._-]*[.]txt".
    • files with paths like ".*/checkstyle.*".
    • files with paths like ".*/checkstyle[.]xml".
  • 77 files match defined criteria (8,119 LOC, 2.6% vs. main code):
    • 28 xml files (5,992 LOC)
    • 39 sh files (1,764 LOC)
    • 1 py files (170 LOC)
    • 4 gitignore files (110 LOC)
    • 1 yaml files (35 LOC)
    • 1 bat files (26 LOC)
    • 1 sbt files (16 LOC)
    • 1 cfg files (4 LOC)
    • 1 gitattributes files (2 LOC)
Other Code
properties
orc
ini
Explore:   circles  |  sunburst

txt22899 LOC (45%) 258 files
json8869 LOC (17%) 47 files
scala8116 LOC (16%) 191 files
java5804 LOC (11%) 128 files
py3276 LOC (6%) 136 files
properties632 LOC (1%) 21 files
md526 LOC (1%) 15 files
xml137 LOC (<1%) 1 file
orc22 LOC (<1%) 1 file
ini18 LOC (<1%) 1 file
avsc9 LOC (<1%) 2 files

scope details...
  • The following criteria are used to filter files:
    • files with paths like ".*[.]properties".
    • files with paths like ".*[.]md".
    • files with paths like ".*/README[.][a-z0-9]+".
    • files with paths like ".*[.]json".
    • files with paths like ".*[.]txt".
    • files with paths like ".*[.](rst|rest|resttxt|rsttxt)".
    • files with paths like ".*/vendor/.*".
    • files with paths like ".*/[Ee]xamples/.*".
    • files with paths like ".*[.]ini".
  • 801 files match defined criteria (50,308 LOC, 15.9% vs. main code):
    • 258 txt files (22,899 LOC)
    • 47 json files (8,869 LOC)
    • 191 scala files (8,116 LOC)
    • 128 java files (5,804 LOC)
    • 136 py files (3,276 LOC)
    • 21 properties files (632 LOC)
    • 15 md files (526 LOC)
    • 1 xml files (137 LOC)
    • 1 orc files (22 LOC)
    • 1 ini files (18 LOC)
    • 2 avsc files (9 LOC)
About This Analysis
learn more about scopes (main, test, generated, build, other)...

  • The main category contains all manually created source code files that are being used in the production.
  • Files in the main category are used as input for other analyses: logical decomposition, concerns, duplication, file size, unit size, and conditional complexity.
  • Test source code files are used only for testing of the product. These files are normally not deployed to production.
  • Build and deployment source code files are used to configure or support build and deployment process.
  • Generated source code files are automatically generated files that have not been manually changed after generation.
  • While a source code folder may contain a number of files, we are primarily interested in the source code files that are being written and maintained by developers.
  • Files containing binaries, documentation, or third-party libraries, for instance, are excluded from analysis. The exception are third-party libraries that have been changed by developers.

learn more about used code analyzers...
  • *.scala files are analyzed with ScalaAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • Unit size analysis
    • Conditional complexity analysis
    • Basic heuristic dependency analysis
  • *.java files are analyzed with JavaAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • Unit size analysis
    • Conditional complexity analysis
    • Advanced heuristic dependency analysis (based on package names)
  • *.py files are analyzed with PythonAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • Unit size analysis
    • Conditional complexity analysis
    • Basic heuristic dependency analysis
  • *.js files are analyzed with JavaScriptAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • Unit size analysis
    • Conditional complexity analysis
    • No dependency analysis
  • *.g4 files are analyzed with DefaultLanguageAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Basic code cleaning (empty lines removed for LOC calculations and duplication calculations)
    • No unit size analysis
    • No conditional complexity analysis
    • No dependency analysis
  • *.html files are analyzed with HtmlAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • Unit size analysis
    • Conditional complexity analysis
    • Advanced heuristic dependency analysis
  • *.css files are analyzed with CssAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • No unit size analysis
    • No conditional complexity analysis
    • No dependency analysis
  • *.thrift files are analyzed with ThriftAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • No unit size analysis
    • No conditional complexity analysis
    • No dependency analysis
  • *.xml files are analyzed with XmlAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • No unit size analysis
    • No conditional complexity analysis
    • No dependency analysis
  • *.bash files are analyzed with ShellAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • No unit size analysis
    • No conditional complexity analysis
    • No dependency analysis
  • *.ps1 files are analyzed with DefaultLanguageAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Basic code cleaning (empty lines removed for LOC calculations and duplication calculations)
    • No unit size analysis
    • No conditional complexity analysis
    • No dependency analysis
  • *.in files are analyzed with RustAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • Unit size analysis
    • Conditional complexity analysis
    • No dependency analysis
  • *.c files are analyzed with CStyleAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • Unit size analysis
    • Conditional complexity analysis
    • No dependency analysis
  • *.yaml files are analyzed with YamlAnalyzer:
    • All basic standard analyses supported (source code overview, duplication, file size, concerns, findings, metrics, controls)
    • Advanced code cleaning (empty lines and comments removed for LOC calculations, additional cleaning for duplication calculations)
    • No unit size analysis
    • No conditional complexity analysis
    • No dependency analysis