Internals

Contents

• Internals
  • On OCaml
  • Architecture
  • Subsystems documentation
  • Extending pfff
    • Adding a new language

Internals

On OCaml

OCaml may seem to be a weird choice. It's not a very popular language. Nevertheless in the "static analysis world", it is one of the most popular languages. Researchers in Berkeley and Stanford are using OCaml for performing analysis of C code (e.g. Necula with CIL at Berkeley, Aiken with Saturn at Stanford), some type inferers for Ruby are written in OCaml (DRuby), the Windows Device verifiers (based on work from Microsoft Research by Thomas Ball on SLAM) is written in OCaml, people in the Linux kernel are using the Coccinelle tool, also written in OCaml, to perform some complex refactoring and to find bugs. In fact one of the latest programming languages from Microsoft, F#, is directly inspired by OCaml.

OCaml is good for code analysis. Trust me.

• OCaml tutorial
• Commons library manual

Architecture

Overview

Subsystems documentation

• Visualizer
• Treemap library
• Parsing PHP
• Analyze PHP

Extending pfff

Adding a new language

You can use the code in lang_lisp/ (the simplest language) as a source of inspiration. Then do in order:

• a simple lexer (in lang_xxx/parsing/lexer_xxx.mll)

  • skeleton

    http://github.com/facebook/pfff/commit/270df0000a13b9cbb24b60ae335421e431285211

  • a -tokens_xxx command line action

    http://github.com/facebook/pfff/commit/dd50723c307fd1ba5868fd4004d2d88feaa423e5

    http://github.com/facebook/pfff/commit/fe5281af39e5bffd5181bc19de13981e45d588e1

  • skeleton for -parse_xxx

    http://github.com/facebook/pfff/commits/master

• an AST with just the tokens (in lang_xxx/parsing/ast_xxx.ml)

  http://github.com/facebook/pfff/commit/dd50723c307fd1ba5868fd4004d2d88feaa423e5

  http://github.com/facebook/pfff/commit/fe5281af39e5bffd5181bc19de13981e45d588e1

• a token-based highlighter => instant gratification (in lang_xxx/analyze/highlight_xxx.ml and few modifications in visual/parsing2.ml)

  • mostly keywords highlighting a la emacs

    http://github.com/facebook/pfff/commit/e0fb08fac390769c471b1453ee7566ce3b6ebd65

  • highlight entities (function definitions, globals)

    http://github.com/facebook/pfff/commit/279be8b22691940311ed49ade088efeaf17a9606

• a simple grammar and parser (in lang_xxx/parsing/parser_xxx.mly)

• an ast-based highlighter => can convey more information (again in lang_xxx/analyze/highlight_xxx.ml)

• a simple global analysis to compute the most important functions (in lang_xxx/analyze/database_light_xxx.ml)

• a semantic-based highlighter => can convey even more information (in visual/analyze/highlight_xxx.ml)