Design: Pipeline Graph Path names

[geekscape]

The Pipeline Graph Path names proposal is in development

Proposed PipelineDefinition

Use a "path:" prefix to indicate that this is a Graph Path name and not an actual PipelineElement.

A bonus is that a similar syntax can be used for other upcoming requirements for adding features to the Graph Path elements

"graph": [
  "(path:summarize TextReadFile TextSummarize TextWriteFile)",
  "(path:translate TextReadFile TextTranslate TextWriteFile)"
]

Alternate suggestion: 1

Use a dictionary with the Graph Path name as the key and the Graph Path PipelineElements as the value.

"graph": [
  "((summarize: (TextReadFile TextSummarize TextWriteFile))",
  "((translate: (TextReadFile TextTranslate TextWriteFile))"
]

• From a visual perspective ... this looks okay ... and only slightly more involved than the proposal (above)

• From an implementation perspective ... this conflicts with Graph Paths using a dictionary to specify Graph PipelineElement properties

  • This is currently used for mapping PipelineElement input / output names when they aren't aligned, e.g an output name doesn't match an input name
  • This can be worked around, but perhaps not worth the effort ?

JSON
   "graph": ["(RandomIntegers Add (random: i))"],

Python
   def _add_node_properties(self, node_name, properties, predecessor_name):

   node_heads, node_successors = Graph.traverse(definition.graph, self._add_node_properties)

Alternate suggestion: 2

Update Graph Paths to entirely be just an S-Expression, rather than the current JSON Graph list of Graph Path S-Expressions (yes, that is how the Graph is currently defined 😮)

"graph": "(((summarize: (TextReadFile TextSummarize TextWriteFile)) (translate: (TextReadFile TextTranslate TextWriteFile)))"

• From a visual perspective ... this will be a mess ...

  • The S-Expression will start and end with 3 curly brackets ((( ,,, ))) and there will be many more in-between
  • Since JSON doesn't permit line breaks within strings, then the Graph much be a single String wrapping past the line width
  • Even relatively simple Graph Path S-Expressions will be hard to read and error prone when writing them ... especially for newcomers

• From an implementation perspective, it will not be too difficult

  • However, it would be more preferable to switch from a combination of JSON / S-Expressions to just an S-Expression ... and that is just too much work right now (benefit versus cost) and too disruptive for customer projects

Thoughts, questions, comments and suggestions welcome !