Define chain rules for complex functions #167

ettersi · 2020-06-02T14:47:22Z

Adds a rigorous definition of how frule and rrule are to be implemented for complex functions.

oxinabox · 2020-06-02T16:13:47Z

Is this the outcome of #159 ?
I have not caught up with it yet.

ettersi · 2020-06-03T01:00:43Z

I would say it's the outcome of https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317 and JuliaDiff/ChainRules.jl#196. However, these are open-ended discussions where it can be hard to determine whether or not consensus has been reached. One of the goals of this PR is to make sure that this discussion does eventually come to a conclusion.

docs/src/FAQ.md

Co-authored-by: willtebbutt <[email protected]>

docs/src/FAQ.md

Co-authored-by: willtebbutt <[email protected]>

…s parts

nickrobinson251 · 2020-06-13T11:58:00Z

It'd be nice to be able to see the rendered version of this addition. But for some reason, the preview docs are not being built (http://www.juliadiff.org/ChainRulesCore.jl/preview-PR167) -- anyone know why?

docs/src/FAQ.md

simeonschaub

Thanks for doing this! I left some comments where I think some things can be clarified a bit. Otherwise this is great!

simeonschaub · 2020-06-16T16:36:13Z

docs/src/FAQ.md

+
+## How do chain rules work for complex functions?
+
+`ChainRules.jl` follows the convention that `frule` applied to a function ``f(x + i y) = u(x,y) + i \, v(x,y)`` with perturbation ``\dot x + i \dot y`` returns the value and 


I think that is too general of a statement. The way you describe it only really applies for frule if your input is real or rrule if your eventual output is real. I find it quite useful to think of forward-mode AD as propagating a perturbation in the input and getting out a derivative wrt to that input and reverse-mode as propagating a sensitivity in the output and getting out a vector of directional derivatives. Those perturbations/sensitivities can be complex, but not the input/output itself.

The point of this FAQ is to point out two things:

frule and rrule are always defined (e.g. even for non-holomorphic Complex -> Complex functions).

These definitions are sufficient to compute all quantities of interest.

I've added a bit more details to the documentation, I hope this helps to clarify.

I guess your comment arose because this convention for complex frule and rrule leads to a really awkward API for e.g. extracting Jacobians. This is definitely an issue, but the right place to fix this is probably in another package built on top of ChainRules, not in ChainRules itself.

These definitions are sufficient to compite all quantities of interest.

Depends on what you are interested in. Like I said above, only if the input/output is real will you get a meaningful result. For C -> C you would either want the Wirtinger primal and conjugate or a 2x2 Jacobian, which we don't currently support. We should clarify that, so that people don't get any wrong impressions.

I have added a note at the very end which I hope addresses your concern. Let me know what you think.

simeonschaub · 2020-06-16T16:42:07Z

docs/src/FAQ.md

+\tfrac{\partial u}{\partial x} \, \dot x + \tfrac{\partial u}{\partial y} \, \dot y + i \, \Bigl( \tfrac{\partial v}{\partial x} \, \dot x + \tfrac{\partial v}{\partial y} \, \dot y \Bigr)
+,
+```
+and similarly `rrule` applied to the same function evaluates the function and returns a pullback which, when applied to adjoint ``\bar u + i \bar v``, returns


I think \bar u is a bit confusing notation here, because it's usually used for conjugation. What do you think of \mathrm{d}u instead, since you are basically pulling back differential one-forms? The notation should probably also be explained briefly.

Good point! I replaced both \bar and \dot with \Delta. This notation (and the earlier dot and bar notation) are documented further up on this FAQ page.

I don't think it makes sense to use \Delta for both, frule and rrule, because one lives in the tangent plane, whereas the other one lives in the cotangent plane.

It's the notation proposed by the ChainRules documentation. Waiting for input from others.

docs/src/FAQ.md

sethaxen

Thanks for taking point on this! Most of my comments are notational. Would it also be useful to address the oft-debated conjugation convention implied by these definitions? When representing all complex numbers as vectors of real numbers, it's difficult at least for me to work out how that relates to the two different discussed conjugation conventions. Put another way, how do these conventions relate to those used by FDM and Zygote (e.g. as documented here: #159 (comment))

docs/src/FAQ.md

Co-authored-by: Seth Axen <[email protected]>

ettersi · 2020-06-22T09:13:15Z

I believe I resolved all the requested changes, so AFAICT this is ready to be merged.

Also, I deleted the multidimensional example again, for several interrelated reasons:

It requires quite a bit of brain power to grasp what's going on, but doesn't help much in terms of clarifying the ideas (I think the 1d case is sufficient for this).
It may wrongly suggest that it rigorously defined complex differentiation in full generality, which it doesn't (see e.g. sethaxen's matrix product example, which isn't really covered ).

sethaxen

Looks great to me! Thanks again!

MasonProtter

Looking good to me.

So if I understand correctly, this is saying we do not take the conjugate in reverse diff, right? Is that direction that we want ChainRules to take? I don't have a super strong feeling either way, though @MikeInnes might. Either way, we need to make it much more explicit which convention we follow.

docs/src/FAQ.md

MasonProtter · 2020-06-22T23:29:44Z

docs/src/FAQ.md

+\Delta u \, \tfrac{\partial u}{\partial x} + \Delta v \, \tfrac{\partial v}{\partial x} + i \, \Bigl(\Delta u \, \tfrac{\partial u }{\partial y} + \Delta v \, \tfrac{\partial v}{\partial y} \Bigr)
+.
+```
+If we interpret complex numbers as vectors in ``\mathbb{R}^2``, then these rules correspond to multiplication with the (transposed) Jacobian of ``f(z)``, i.e. `frule` corresponds to


Suggested change

If we interpret complex numbers as vectors in ``\mathbb{R}^2``, then these rules correspond to multiplication with the (transposed) Jacobian of ``f(z)``, i.e. `frule` corresponds to

If we interpret complex numbers as vectors in ``\mathbb{R}^2``, then the `frule` (`rrule`) corresponds to multiplication with the (transposed) Jacobian of ``f(z)``, i.e. `frule` corresponds to

👍, but "the frule" sounds weird to my (nonnative) ear. Removed the "the".

sethaxen · 2020-06-23T00:11:15Z

So if I understand correctly, this is saying we do not take the conjugate in reverse diff, right? Is that direction that we want ChainRules to take? I don't have a super strong feeling either way, though @MikeInnes might. Either way, we need to make it much more explicit which convention we follow.

Unless I have misunderstood something, the convention proposed here is the same as Zygote's and FiniteDifference's. @ettersi I thought it was clear in the matrix section earlier, but I agree it is less clear now what the convention is. Maybe it makes sense to bring it back?

Co-authored-by: Mason Protter <[email protected]>

ettersi · 2020-06-23T00:54:23Z

the derivative part of rrule can be implemented as conj(f'(z)) \Delta f

Doesn't this address the question about the conjugate? It's only the scalar case, but I guess it is clear that if we conjugate scalars than we also conjugate matrices.

Btw, thanks to @MasonProtter for starting Taking Complex Autodiff Seriously in ChainRules. This is what got me started on this project.

docs/src/FAQ.md

MasonProtter · 2020-06-23T05:12:17Z

I think maybe we should add a sentance pointing out explicitly that the Jacobian matrix shown in the rrule is transposed relative to the Jacobian for frule and that transpose is what encodes the complex conjugate.

The fact that I missed that difference after explicitly looking for it makes me thing I might not be the only one.

Thanks a lot for writing this up @ettersi!

ettersi · 2020-06-23T05:38:08Z

I think maybe we should add a sentance pointing out explicitly that the Jacobian matrix shown in the rrule is transposed relative to the Jacobian for frule and that transpose is what encodes the complex conjugate.

I changed the rrule formula to ^T instead of writing out the transpose explicitly. Hope this makes it clearer.

sethaxen · 2020-06-24T17:22:08Z

@ettersi is this ready to merge? And does anyone else have objections to merging?

docs/src/FAQ.md

Co-authored-by: Mason Protter <[email protected]>

ettersi · 2020-06-25T09:50:53Z

I'm happy for this to be merged! 🥳

sethaxen · 2020-06-25T16:28:42Z

Okay! Unless there are any objections before then, I'll merge tomorrow.

oxinabox · 2020-06-25T20:51:40Z

and with that like over a dozen PRs will become unblocked!

sethaxen · 2020-06-25T21:27:58Z

and with that like over a dozen PRs will become unblocked!

Or at least get closer to being unblocked! It's probably best to make the existing rules conform to these new standards before merging PRs with new rules. Then there are also open questions like https://github.com/JuliaDiff/ChainRules.jl/issues/212 that we haven't had to address until now.

oxinabox · 2020-06-26T16:16:58Z

Thank you everyone involved in this, its been a journey!

Define chain rules for complex functions

024dc29

ettersi mentioned this pull request Jun 4, 2020

Custom Covector Wrapper #165

Open

oxinabox reviewed Jun 12, 2020

View reviewed changes

docs/src/FAQ.md Outdated Show resolved Hide resolved

willtebbutt reviewed Jun 12, 2020

View reviewed changes

docs/src/FAQ.md Outdated Show resolved Hide resolved

oxinabox and others added 2 commits June 12, 2020 17:44

use dots not Deltas

4f9426d

Co-authored-by: willtebbutt <[email protected]>

make FAQ a question

0963d16

willtebbutt reviewed Jun 12, 2020

View reviewed changes

docs/src/FAQ.md Outdated Show resolved Hide resolved

willtebbutt reviewed Jun 12, 2020

View reviewed changes

docs/src/FAQ.md Outdated Show resolved Hide resolved

oxinabox and others added 2 commits June 12, 2020 22:05

say pullback

c063297

Co-authored-by: willtebbutt <[email protected]>

Add expressions for multiplication with Jacobian, and LaTeXify variou…

f1562eb

…s parts

oxinabox reviewed Jun 16, 2020

View reviewed changes

docs/src/FAQ.md Outdated Show resolved Hide resolved

be as precise for frule as for rrule

a233860

simeonschaub reviewed Jun 16, 2020

View reviewed changes

Simon Etter added 6 commits June 17, 2020 14:15

Delete trailing whitespaces

a9b577a

Add "the"

36875b6

Replace \dot and \bar with \Delta

a98c0b1

Fix typo

a8aafd6

Minor language tweaks

73b2b27

Add code examples

5c3ffca

ettersi mentioned this pull request Jun 17, 2020

Chunked rules for complex derivatives #168

Closed

Split code blocks

40166b8

nickrobinson251 added the Complex Differentiation Relating to any form of complex differentiation label Jun 17, 2020

sethaxen requested changes Jun 17, 2020

View reviewed changes

ettersi and others added 4 commits June 18, 2020 08:55

Remove \,

80f97af

Co-authored-by: Seth Axen <[email protected]>

Remove stray rrule(...)[2]

cd20b31

Co-authored-by: Seth Axen <[email protected]>

Remove stray rrule(...)[2]

521e72f

Replace x -> z where necessary

06b9d50

Simon Etter added 2 commits June 22, 2020 17:01

Delete multidimensional case

6ef79f8

Minor improvements in Note.

62347f6

sethaxen mentioned this pull request Jun 22, 2020

Conjugation convention changed from v0.9 to v0.10 JuliaDiff/FiniteDifferences.jl#87

Closed

sethaxen approved these changes Jun 22, 2020

View reviewed changes

This was referenced Jun 22, 2020

Change convention for complex differentiation JuliaDiff/ChainRules.jl#210

Closed

Conjugate partials in scalar rrule #170

Merged

MasonProtter reviewed Jun 22, 2020

View reviewed changes

pullback -> pullback function

269adf0

Co-authored-by: Mason Protter <[email protected]>

Minor clarification

2f13561

MasonProtter reviewed Jun 23, 2020

View reviewed changes

docs/src/FAQ.md Outdated Show resolved Hide resolved

Change rrule formula to ^T

252c0e5

MasonProtter mentioned this pull request Jun 23, 2020

Correct chainrules for abs2, abs, conj and angle JuliaDiff/ChainRules.jl#196

Merged

MasonProtter reviewed Jun 24, 2020

View reviewed changes

docs/src/FAQ.md Show resolved Hide resolved

MasonProtter reviewed Jun 24, 2020

View reviewed changes

docs/src/FAQ.md Outdated Show resolved Hide resolved

docs/src/FAQ.md Outdated Show resolved Hide resolved

ettersi and others added 2 commits June 25, 2020 17:48

Change df_dz and friends to du_dx and friends

f1921e9

Co-authored-by: Mason Protter <[email protected]>

Change df_dz and friends to du_dx and friends

7997aa8

Co-authored-by: Mason Protter <[email protected]>

sethaxen merged commit d187432 into JuliaDiff:master Jun 26, 2020

sethaxen mentioned this pull request Jun 28, 2020

RFC: Rules for real-to-complex and complex-to-real functions #176

Open

ettersi deleted the patch-1 branch July 13, 2020 08:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Define chain rules for complex functions #167

Define chain rules for complex functions #167

ettersi commented Jun 2, 2020

oxinabox commented Jun 2, 2020

ettersi commented Jun 3, 2020

nickrobinson251 commented Jun 13, 2020

simeonschaub left a comment

simeonschaub Jun 16, 2020

ettersi Jun 17, 2020

ettersi Jun 17, 2020

simeonschaub Jun 17, 2020

ettersi Jun 18, 2020

simeonschaub Jun 16, 2020

ettersi Jun 17, 2020

simeonschaub Jun 17, 2020

ettersi Jun 17, 2020

sethaxen left a comment

ettersi commented Jun 22, 2020

sethaxen left a comment

MasonProtter left a comment

MasonProtter Jun 22, 2020

ettersi Jun 23, 2020

sethaxen commented Jun 23, 2020 •

edited

Loading

ettersi commented Jun 23, 2020 •

edited

Loading

MasonProtter commented Jun 23, 2020 •

edited

Loading

ettersi commented Jun 23, 2020

sethaxen commented Jun 24, 2020

ettersi commented Jun 25, 2020

sethaxen commented Jun 25, 2020

oxinabox commented Jun 25, 2020

sethaxen commented Jun 25, 2020

oxinabox commented Jun 26, 2020


		## How do chain rules work for complex functions?

		`ChainRules.jl` follows the convention that `frule` applied to a function ``f(x + i y) = u(x,y) + i \, v(x,y)`` with perturbation ``\dot x + i \dot y`` returns the value and

	If we interpret complex numbers as vectors in ``\mathbb{R}^2``, then these rules correspond to multiplication with the (transposed) Jacobian of ``f(z)``, i.e. `frule` corresponds to
	If we interpret complex numbers as vectors in ``\mathbb{R}^2``, then the `frule` (`rrule`) corresponds to multiplication with the (transposed) Jacobian of ``f(z)``, i.e. `frule` corresponds to

Define chain rules for complex functions #167

Define chain rules for complex functions #167

Conversation

ettersi commented Jun 2, 2020

oxinabox commented Jun 2, 2020

ettersi commented Jun 3, 2020

nickrobinson251 commented Jun 13, 2020

simeonschaub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sethaxen left a comment

Choose a reason for hiding this comment

ettersi commented Jun 22, 2020

sethaxen left a comment

Choose a reason for hiding this comment

MasonProtter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sethaxen commented Jun 23, 2020 • edited Loading

ettersi commented Jun 23, 2020 • edited Loading

MasonProtter commented Jun 23, 2020 • edited Loading

ettersi commented Jun 23, 2020

sethaxen commented Jun 24, 2020

ettersi commented Jun 25, 2020

sethaxen commented Jun 25, 2020

oxinabox commented Jun 25, 2020

sethaxen commented Jun 25, 2020

oxinabox commented Jun 26, 2020

sethaxen commented Jun 23, 2020 •

edited

Loading

ettersi commented Jun 23, 2020 •

edited

Loading

MasonProtter commented Jun 23, 2020 •

edited

Loading