flux errors when parameters contain spaces

[aowen87]

When doing a parameter sweep, we sometimes are sweeping over variables that contain spaces. Maestro will then create launch scripts that contain spaces in them, but it generally handles this just fine. On flux, this falls apart, and Maestro complains that it can't find the file.

Below is a simple reproducer that can be run with the following command on a flux machine:

maestro run foo.yaml -p pgen.py

foo.yaml

batch:
  bank: guests
  host: rzadams
  queue: pdev
  type: flux

description:
  name: study
  description: foo

generator.parameters:
    FOOBAR:
        values: ["--foobar 9 2 5 1", "--foobar 1 6 4 3"]
        label: FOOBAR.%%

study:
  - name: foo-study
    description: foo
    run:
      cmd: |

        echo $foobar

      nodes: 1
      procs: 1
      nested: True # disabling this seemed to fix the problem, but we need this enabled
      walltime: 00:15:00

pgen.py

import sys
from maestrowf.datastructures.core import ParameterGenerator
from sklearn.model_selection import ParameterGrid
import yaml
try:
    from yaml import CLoader as Loader, CDumper as Dumper
except ImportError:
    from yaml import Loader, Dumper

def get_custom_generator(env, **kwargs):
      """
      Create a custom populated ParameterGenerator.
      
      This function recreates the exact same parameter set as the sample LULESH
      specifications. The point of this file is to present an example of how to
      generate custom parameters.
      
      :returns: A ParameterGenerator populated with parameters.
      """
      import sys
      p_gen = ParameterGenerator()

      yaml_file = ""
      for arg in sys.argv:
          if arg.endswith(".yaml"):
              yaml_file = arg

      yml = yaml.load(open(yaml_file).read(), Loader=Loader)
      p_in = {}
      labels = {}

      linked_ps = {}
      linked = {}

      for key, val in yml["generator.parameters"].items():
          if "values" in val:
              if isinstance(val["values"], (list,tuple)):
                  p_in[key] = list(val["values"])
              else:
                  p_in[key] = [val["values"],]

              labels[key] = val["label"]

          # Not sure what "LINKED" is. It was carried over from Charles.
          elif key == "LINKED":
              linked = val

      for plink in linked:
          for link in linked[plink]:
              if not plink in linked_ps:
                  linked_ps[plink] = {}
              linked_ps[plink][link] = p_in.pop(link)

      grid = ParameterGrid(p_in)
      p = {}
      for g in grid:
          for key in g:
              if key not in p:
                  p[key] = [g[key], ]
                  if key in linked_ps:
                      for link in linked_ps[key]:
                          p[link] = [linked_ps[key][link][p_in[key].index(g[key])],]
              else:
                  p[key].append(g[key])
                  if key in linked_ps:
                      for link in linked_ps[key]:
                          p[link].append(linked_ps[key][link][p_in[key].index(g[key])])


      for key, val in p.items():
          labels[key] = labels[key] + "_%%"

      for key, val in p.items():
          p_gen.add_parameter(key, val, labels[key])
      return p_gen

[doutriaux1]

@aowen87 you could probbly use --hashws flag but it leads to uggly names. I also learned recently that rather than
label: FOOBAR.%%

You can pass a list of the labels to use so you could do that as well and it wolud look nicer.

@jwhite242 any suggestion?

[aowen87]

I have a temporary workaround by filling the spaces with underscores and then removing them right before they're used. It's a little clunky, but it gets us moving.

[jwhite242]

Is that comment on the nested key actually correct? very odd that that would fix something like this as this seems like a more general parameter sanitizing issue that should break all schedulers (honestly still not sure what the point of using nested=false is given how resources get counted by flux when you turn it of..).

I'll start working on a patch for that to auto replace spaces with underscores when making both the scripts and the workspace paths. I also definitely recommend charles' suggestion to make a more human friendly id string to put in the labels as double dashes are gross in paths. (similarly, slashes will get underscored so paths as parameter values don't cause other unhelpful side effects)

[aowen87]

Is that comment on the nested key actually correct?

It seemed to have fixed it in one of my runs, but I didn't spend much time verifying that. I thought it might be a useful observation.

Thanks for jumping this!