Design of Pyomo expressions with kernel subclasses

[whart222]

I'm pretty sure that the current design of Pyomo5 expressions won't robustly work with subclasses of the pyomo.core.kernel classes (e.g. variable and expression). I think the same issue likely applies to Coopr3.

There's a fundamental tension in the design of these expression systems.

1. We avoid isinstance() because it is slow
2. We use global sets of class types that are checked to test class properties.

For example, the pyomo5_variable_types set is used to confirm whether a class is a variable type in many expression tree walkers.

But the problem with (2) is that we have various global sets that are static. Thus, if a user creates a new subclass, then we should not expect the core expression tree walkers to work properly.

I see 3 possible solutions to this:

1. Go back to using isinstance()
2. Use a global "registration" for classes. This could be simple done with a class decorator, which has a nice property that not all subclasses in a hierarchy would be treated as valid classes during search.
3. Add ISA methods. For example, is_variable() could be used to identify variable classes.

Option (3) is consistent with our current design, but I'm leaning towards (2). The problem with (3) is that we need to migrate these ISA methods to lots of different classes. I'm still running across subtle bugs where different methods weren't migrated properly to all relevant classes.

Thoughts?

[whart222]

NOTE: I'm experimenting with using a Pyomo issue to track a design discussion. This will provide a nice archive of this design concern. But I'm worried that we limit the discussion to just the folks that are invited. Thoughts? This design concern isn't about the public API, so I think it's OK to involve fewer developers. But ... (?)

[whart222]

A very good argument for going with option (3) is that it doesn't require the developer to know about these global sets. Instead, they just need to look at the ExpressionBase API to see what ISA tests are supported. I think I'm leaning towards (3) now...

[jsiirola]

I am confused... The sets we have now are not static, and get augmented by things like as_numeric (the design is that the first time we come across a new type it may be a little slower, but once we know that it is - for example - a numeric type, then we add it to the set of known numeric types).

The expression system shouldn't need to know if the object is a Var or Param or Expression or whatnot. The actions are completely determined by the is_fixed, is_constant, and potentially_variable methods. Forming an expression shouldn't have anything to do with the data type of the node.

[qtothec]

@whart222: related to your concern that the discussion is limited to just those invited: I pipe in.

I don't have strong feelings as long as there's not a large performance impact and it's clear how things should be used.

[jsiirola]

Side note (since @qtothec brought it up elsewhere): isinstance isn't too slow as long as it returns True. It is significantly slower when it returns False.

[whart222]

@jsiirola The set pyomo5_variable_types is an example of data that is not updated if a user creates a subclass of the kernel variable class. One of the key goals of pyomo.core.kernel was to facilitate extensions like this, our previous designs for the expression system were too static.

[whart222]

@jsiirola I'm aware of the asymmetry of isinstance(), but it's generally safer to assume it will be slow without detailed knowledge of the runtime behavior across a large number of test instances. That's hard information to gather, so I've followed your recommendation to eliminate the use of isinstance() in most cases.

[whart222]

I'm closing this discussion. After drafting this, I realized that option (3) above is the best alternative. It doesn't require registration of subclasses, and this design is consistent with how we're handling similar concerns for testing whether objects are expressions.

[jsiirola]

OK - I suppose this question will just come back up when we review #272.

[whart222]

No. I deleted the offending sets! You can't comment on what's not there!

…

On Dec 6, 2017 10:28 AM, "John Siirola" ***@***.***> wrote: OK - I suppose this question will just come back up when we review #272 <#272>. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#276 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAsb-CKp5KMvxjnul10CBZDuKeO-QwJ9ks5s9s6igaJpZM4Q2fSI> .

[carldlaird]

Can someone clarify how ctypes fit into this discussion? Only for components, so they don't apply in expressions?

[whart222]

FYI, Carl and I discussed the design options using ctypes vs the is_* functions. In principle we could use a design like ctypes to capture the duck typing characteristics in the core, rather than is_* functions. But that option would require the introduction of new data into the expression objects.

[jsiirola]

Originally ctype was only used for grouping components together for reporting. We now use it for interrogating the model as well (e.g., "give me the list of all the constraints in the model"). The real power of ctypes (that arose well after their implementation) is what Bill hinted at: they support user-managed duck typing in the core. That is, as long as a component implements an API consistent with, say, Constraint, and declares that it's ctype is "Constraint", then Pyomo will treat the component as a Constraint (regardless of the class hierarchy). It also supports derived Components (like Disjunct) that don't want to be handled the same way as their base class. For example, our {LP, NL, etc.} writers should happily write out models that contain Blocks, but should not write out models with active Disjuncts, because even through the Disjunct is a Block, the writer doesn't have the ability to express the semantic meaning of "Disjunct" in the NL or LP format.

It is unclear why the expression system would ever need to know the "type" of a leaf node in the tree: in my experience, it is sufficient to know the characteristics of the node (is_constant, is_fixed, potentially_variable, and maybe polynomial_degree).

[whart222]

John: When you're searching through a tree, you need to "detect" that you're at a leaf. In Pyomo, we look at the class type to do that, for example to see if the object is a numeric constant or a variable class.

…

--Bill

On Thu, Dec 7, 2017 at 4:41 PM, John Siirola ***@***.***> wrote: Originally ctype was only used for grouping components together for reporting. We now use it for interrogating the model as well (e.g., "give me the list of all the constraints in the model"). The real power of ctypes (that arose well after their implementation) is what Bill hinted at: they support user-managed duck typing in the core. That is, as long as a component implements an API consistent with, say, Constraint, and declares that it's ctype is "Constraint", then Pyomo will treat the component as a Constraint (regardless of the class hierarchy). It also supports derived Components (like Disjunct) that *don't* want to be handled the same way as their base class. For example, our {LP, NL, etc.} writers should happily write out models that contain Blocks, but should *not* write out models with active Disjuncts, because even through the Disjunct is a Block, the writer doesn't have the ability to express the semantic meaning of "Disjunct" in the NL or LP format. It is unclear why the expression system would ever need to know the "type" of a leaf node in the tree: in my experience, it is sufficient to know the characteristics of the node (is_constant, is_fixed, potentially_variable, and maybe polynomial_degree). — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#276 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAsb-AcrBkkOohWLCbfR3xL1E9qqLWEQks5s-HeegaJpZM4Q2fSI> .

[jsiirola]

Historically, in coopr3 / pyomo4, “e.__class__ in native_types or not e.is_expression()” was sufficient for identifying leaf nodes when you were searching by through a tree - that is, evaluating it and not manipulating it. You needed an extra test if you were manipulating trees and wanted to treat Expression as a leaf.

Why wouldn’t that work here?

[whart222]

There are other types of leaf nodes, including parameter and variable objects. And the sets I was referring to in this issue concerned the efficient identification of those objects.

If my reasoning for closing this issue isn't clear, then let's pull this conversation off-line. I'm pretty sure my comments here are helping you understand the design question I was mulling over.