Expressing dependencies between parameters

[ceball]

(Attempt to capture notes/plan from PR, JB, JLS, CB - feel free to edit and/or comment.)

Users of param GUI widget toolkits often want to be able to control the range or type of some widgets based on the values of others.

For example, consider specification of a date range: one might want to allow a user either to select start date and end date, or to select a start date and duration. Or the value in one numeric widget might affect the range of another slider. Or a widget should be greyed out or not depending on the value of another widget.

Beyond GUI toolkits, being able to express dependencies between parameters seems like something that might be useful to any user of param. Many of us have occasionally felt the need for linking parameters while using param in the real world, but we've never actually needed it enough to justify the added complexity, and have solved our problems a different way.

We should therefore consider at what level we should support expressing dependencies between parameters.

Solve at param level?

It's likely that many of the dependencies desired by GUI widget users could be expressed at the domain level (i.e. using param), so it could be valuable for param to support expressing those dependencies in a general way, such as by having parameters register that they subscribe to other parameters' values (being careful to avoid a vicious circle) and allowing code to be registered to respond to such changes. Solving this at the param level would handle a large class of problems across all the widget toolkits, so could be very beneficial. However, it has the danger of making Param impossible to understand (which data flows where, when?) and maintain, if not done well.

Solve at param GUI toolkit level?

Since it's not yet clear if the above can be done well, or would inherently have the problems described above when sufficiently advanced to cover the typical real-world cases where these issues arise, an alternative solution would be to extend the toolkit-specific classes like paramnb.Widgets() to allow an update method to be registered that gets called whenever a parameter is changed. Such a method would then check for changes in parameters and apply changes to others that depend on it. This seems straightforward to implement, and would allow us to discover exactly what it is we might want to support in param.

However, this solution has several disadvantages over the param-level solution:

• Doesn't help non-gui users and would have to be implemented for each widget toolkit.

• The GUI code would have to know about the underlying objects. Although a convention could be adopted whereby if specific parameter names (e.g. start_date, end_date, duration) are detected, the appropriate widgets and logic are automatically used (ensuring that all three of those values are set consistently regardless of how the data was input by the user).

Proposal

We propose to solve at the GUI toolkit level, gather real-world use cases together as a set of demos, and then consider again about implementing at the param level. We take the risk that we end up just using the GUI level solutions forever, and code based on them spreads everywhere.

Additional note: megawidgets

While implementing the above proposal, note that some things - such as being able to select date ranges flexibly - cut across nearly every application domain, and so it would make sense to encapsulate at the widget level. E.g. date range selection logic could be encapsulated into a reusable megawidget (for each GUI toolkit) that at bottom supplies a concrete date range for Param to consume, via a flexible user interface that could select between all typical options like explicit start and end dates, start date plus duration, end date minus duration, duration up to today, duration up to the most recent start of week/end of week/start of month/end of month, etc.

Advantages: fully client side (and thus responsive e.g. considering case of JS widgets communicating with python over the internet) and would prevent downstream code from having to worry about any of these issues. For anything covered by such a megawidget, the underlying Param-based code only needs to express that it needs a date range, and the toolkits will handle the rest. I.e. using megawidgets could cut down on the complexity of parameter dependency/linking support required.

Disadvantages: Would need to be implemented for each widget toolkit, more or less from scratch. Jim already proposed a simpler version of this for ipywidgets (jupyter-widgets/ipywidgets#1118), but it was rejected as best being implemented as a separate library, so it might be a struggle to get each such library to accept responsibility for this functionality.

[ceball]

JAB:

In general, I want to strongly argue for a design where one can specify the linkage between arbitrary parameters and the code that generates a HoloViews object, in a way that does not require or depend on any widget-related code. I.e., everything should be expressible at the Param level, constructing a Parameterized object that has (a) parameters, (b) methods that read those parameters and construct something, and (c) some way to declare which specific parameters (or by default, all parameters as a group) a given method depends on. That way a Widgets instance can re-run that method on the underlying Parameterized object when one of those parameters changes, without having to have any knowledge about what that particular class is doing. I think this is an extremely general pattern that safely separates the domain-specific aspects (that a specific set of parameters is used by a specific method to generate some object) from anything to do with GUIs, widgets, or user interaction. Sure, here we want to use that declaration for widgets and for HoloViews, but it's a very general notion and something that could end up useful in many different scenarios (e.g. partially reevaluating a complex computational workflow).

[philippjfr]

(c) some way to declare which specific parameters (or by default, all parameters as a group) a given method depends on

@jbednar Could you clarify what you mean by this in practice? Do we really want to declare what parameters a method depends on all the time? What happens if you forget to declare it? It seems this is bound to increase the complexity of param as well as complicating the declaration of a Parameterized class.

Having not thought about this too deeply my initial suggestion is simply to introduce set_hooks, which will let a user set up whatever dependencies they want between parameters without having to introduce some complex API for declaring dependencies in some special way. I'm happy to be persuaded of some other solution but based on your description I currently cannot imagine what that would look like.

[ceball]

Do we really want to declare what parameters a method depends on all the time?

I think the suggestion is for the option to say that a particular method depends on (some) parameters, not for it to be required to do so.

my initial suggestion is simply to introduce set_hooks, which will let a user set up whatever dependencies they want between parameters without having to introduce some complex API for declaring dependencies in some special way

I agree param should be kept simple, but I also think it's important for param not to suggest a way of doing things that leads to complicated/varied solutions by users - if possible!

Sorry for no details yet. I have been thinking about it in the background but can't say what exactly I propose yet.

[jbednar]

There would be absolutely no requirement that one would have to declare that a method depends on a Parameter, no more than there is a requirement that you declare docs, bounds, etc. If you do declare those things, then other code can make use of this additional information, but if you don't, it can't, and so be it. We can probably make the default behavior simply to re-run the relevant method whenever any parameter changes, leaving such declarations only for cases when someone wants to express finer control over which bit to re-run.

I'm happy for set_hook to end up being part of a solution, but it is vastly preferable for the domain-specific (only Param-based) code to be able to express the domain-specific information, that can then be processed in a general, domain-independent way by the widget and GUI code. E.g. maybe we can automatically install a domain-independent, brainless set_hook on the relevant parameters based on information declared in the Parameterized class, which would be much better than to have something external that is deeply tied up in which methods and parameters interact. That sort of logic is hard enough to work out in the context of a single Parameterized class hierarchy, let alone some separate shadow widget-based class that will then drift apart from what the Parameterized declares (since by definition it's declared and maintained separately from the Parameterized).

If we can achieve this separation, we'll have a hope of maintaining things in the long run. If not, we'll take the hit, but I don't want to take it just because we haven't tried to do it properly first.

[philippjfr]

Just to clarify I didn't think there would be any requirement to always have to express dependencies I just have trouble envisioning in what cases you would declare such dependencies, in what cases you wouldn't, how you would express them or indeed what it actually means to declare dependencies on specific methods. I'm just trying to get an idea of what your actual proposal is.

Using a set_hook I can clearly envision how to declare a dependency, e.g. I can simply declare a set_hook on parameter A, which modifies parameter B and I can do this at any time even after declaring the class itself. Expressing this kind of thing as methods on the class sounds to me like it could get messy and result in Parameterized classes that are mostly concerned with setting up these dependencies but maybe that's preferable over declaring functions. Based on your description I'm imagining something like this:

class Example(param.Parameterized):

    A = param.Number(default=10)

    B = param.Number(default=5)

    @link('A')
    def link_a_to_b(self, **parameters):
        self.B = parameters['A']/2.

The equivalent using set_hooks would look something like this:

class Example(param.Parameterized):

    A = param.Number(default=10)

    B = param.Number(default=5)
    
    def __init__(self, **params):
        super().__init__(**params)
        self.add_hooks({'A': lambda A: setattr(self, 'B', A/2.)})

Is this anywhere close to what you were imagining?

[jlstevens]

Also not thinking too deeply about it, here is my suggestion:

class Example(param.Parameterized):

    A = param.Number(default=10)

    B = param.Number(default=5)

    @param.dependencies([A],[B])
    def link_a_to_b(self, **parameters):
        self.B = parameters['A']/2.

The only difference is 1) no need to declare parameters as strings as the actual objects should be in the class namespace (and we always write our methods after our parameter declarations) and 2) it is nice to declare both the inputs (parameter reads) and outputs (parameter writes) together.

[jlstevens]

Of course this could also be supported simultaneously as the decorator that declares reads and writes together:

class Example(param.Parameterized):

    A = param.Number(default=10)

    B = param.Number(default=5)

    @param.reads([A])
    @param.writes([B])
    def link_a_to_b(self, **parameters):
        self.B = parameters['A']/2.

Probably better if you do only reads or only writes - otherwise, I don't like the wasted vertical whitespace.

[ceball]

We've implemented what we think we need. See #262 for any leftover issues.

[jl-massey]

None of the trivial examples seem to capture the most obvious use of dependent parameters. The most trivial example I can think of to demonstrate the point is to have a dropdown list of countries, with another list that needs to contain the states/territories based on one or more selected countries. This SHOULD be trivial (and is for most implementations), but it seems impossible in hv.
Am I overlooking something obvious?

[philippjfr]

@jl-massey The widgets in holoviews long precede any ability to express dependent parameters. We are currently building a new library called panel which, among many other things, will make it easy to set up dependencies between widgets with or without the use of param dependencies.