MCore full bootstrap

[johnwikman]

The goal with a full bootstrap is to support the current code as is without any major rewrites (might need to correct some bugs...).

Areas identified for bootstrapping MCore:

• Lexer & Parser
  • Create a LR(1) parser generator for The Tool
  • Add support for factoring out regex's in the tool; to make supporting #var syntax less horrible
  • Update the existing mcore.syn to be LR(1) compliant and support #var and co.
  • Make glue code to translate between the parsed/generated AST and the intended AST
    • Including some well-formedness checks, such as that syn's and sem's does not appear at top-level
• MLang translation to MExpr
  • MLang AST in MCore
  • Symbolize
  • Demand driven generation of code definition sem's and syn's (analysis probably done in symbolize)
  • Implement merging of fragment
    • Pattern analysis (ordering by subset of values matched (least first), conflict checking)
    • Force constructor names to be distinct in syn's
  • Include handling
• Dead code elimination (as part of symbolize?)
• Ensure that external's are fully applied and that names of externals are globally unique (among externals)

Unclarities:

• Pruning of external utests: Necessary or not?
  • Test with current setup to never run utest pruning, see if we can come to a conclusion earlier

Delimitations (things we would like to do, but not perform now):

• Include revamp (namespacing, etc.)
• Arbitrary top-level definitions inside lang fragments (needs to be discussed)
  • E.g., can we have an include inside a language fragment?
• Renaming of syn's and sem's (between MLang fragments)
• Reevaluate duplicate definitions of constructors in syn's
• Prune the use of Obj.magic in OCaml codegen
  • Defunctionalization of higher-order function arguments
• Parsing comments as AST nodes
  • Doxygen/docstring support
• Plan to shrink and kill boot
• Make tool generate into the AST that we directly use

Dependency graph

(Grey = done)

digraph G {
  node[style=filled, color="lightgrey"];
  lrk[label="LR(k) Parser"];
  implang[label="Implement MLang AST \n in separate mlang/ast.mc"];
  mlang_pprint[label="MLang pprint"];

  node[style="", color="black"];
  lrk_runerr[label="Good \"parse\" \n errors (runtime)"];
  lrk_tblerr[label="Sufficient LR(k) \n table gen error info"];
  lrk_opt[label="LR(k) optimization"];
  include_handl[label="Include handling"];
  include_syn[label=".syn include handling"];
  symbol_handl[label="MLang Symbolizer \n (use, fragment, namespaces)"];
  pattern_handl[label="Pattern analysis"];
  frag_handl[label="Language composition \n (merging fragments)"];
  lang_itemdetect[label="Extinct language \n item detection"];
  mlang_lower[label="MLang lowering \n to MExpr"];
  hashstring[label="Update handling of \n hashstring in lexer"];
  integrate_tool[label="tool.mc integration \n with LR(k) parser"];
  spec_tool[label="specifying \n Mcore in tool.mc"];
  trans_ast[label="Translating to \n {mlang,mexpr}/ast.mc"];
  tool_lrkerrs[label="integrate better LR(k) \n table errors with tool.mc"];
  
  mexpr_pprint[label="MExpr deadcode \n elimination (David)"];
  check_externalapp[label="Check that externals \n are fully applied"];

  full_bootstrap[label="Full bootstrap"];

  lrk -> lrk_runerr;
  lrk -> lrk_tblerr;
  lrk -> lrk_opt;
  lrk -> implang;
  
  implang -> mlang_pprint;
  implang -> symbol_handl;
  implang -> pattern_handl;
  pattern_handl -> frag_handl;
  symbol_handl -> frag_handl;
  symbol_handl -> lang_itemdetect;
  lang_itemdetect -> mlang_lower;
  frag_handl -> mlang_lower;
  symbol_handl -> mlang_lower;
  
  
  mlang_pprint -> integrate_tool;
  
  integrate_tool, lrk_tblerr -> tool_lrkerrs;
  
  lrk -> hashstring;
  
  hashstring -> spec_tool;
  integrate_tool -> spec_tool;
  spec_tool, implang -> trans_ast;

  symbol_handl, trans_ast -> include_handl;
  include_handl -> include_syn;
  integrate_tool -> include_syn;
  
  trans_ast -> full_bootstrap;
  include_handl -> full_bootstrap;
  mlang_lower -> full_bootstrap;
  include_syn -> full_bootstrap;
  check_externalapp -> full_bootstrap [style=dashed];
  mexpr_pprint -> full_bootstrap [style=dashed];
  tool_lrkerrs -> full_bootstrap [style=dashed];
  lrk_runerr -> full_bootstrap [style=dashed];
}

Dependency Graph (graphviz)