Improve performance of calc_connected_components

[mtanneau]

See #913 and profile therein. This PR focuses on improving the performance of calc_connected_components by implementing a more efficient graph search and identification of connected components.

In particular, a significant amount of time is (currently) spent in this one line:

PowerModels.jl/src/core/data.jl

Line 2538 in 8e346b4

ccs = (Set(values(component_lookup)))

In the case where there is a single connected component, this ends up computing the union of N sets, all of which are identical and contain all the buses in the network. This is because the current implementation stores the entire connected component for every bus in the system, which also comes at a O(n^2) memory cost.

I replace the previous DFS search with a more memory-efficient search that indexes each connected component with the smallest bus index in that connected component. There is a O(n log n) cost for sorting bus IDs, and then a O(E) cost for the graph search. The memory cost is O(n). It also saves time because it avoids the costly computation of a set of (large) sets. Note that I kept the last step (converting to a Set{Set{Int}}) to comply with the previous function definition.

Timings of make_basic_network after the changes: times are improved, the impact on memory is marginal

Case	#buses	old time (s)	new time (s)	old memory (MiB)	new memory (MiB)
1354_pegase	1354	0.21	0.30	28.5	28.9
2869_pegase	2869	0.62	0.43	66.8	65.3
9241_pegase	9241	2.50	1.67	208.4	204.2
13659_pegase	13659	5.72	2.47	303.1	306.2

and corresponding profile:

[mtanneau]

@ccoffrin I believe we can also remove the _cc_dfs function. I didn't because I was afraid it might break something downstream.

Side note: Graphs.jl has a lot of functionality to do these kind of graph operations, which are likely more efficient (at the cost of yet another dependency).

[ccoffrin]

Can you remind me how this Julia && syntax works? I think it means, if the lhs is true then execute the rhs?

[mtanneau]

This is equivalent to

if component_lookup[i] != i0
    continue
end

Julia evaluates boolean expressions lazily. Thereby, given A && B, if A evaluates to false, then B is not evaluated.

[odow]

My rule is that you should always write out the if end version. It's much simpler to read.

[ccoffrin]

That's my preference as well, if you don't mind @mtanneau.

[ccoffrin]

Can you remind me how this Julia || syntax works? I am guessing that it means, if the lhs is false then execute the rhs?

Is it also the case that component_lookup[k] == i if it does not equal i0, if not, then we have an issue where a previous labeling did not explore the graph correctly, right?

[mtanneau]

The syntax is equivalent to

if component_lookup[k] == i0
    # do nothing
else
    continue
end

[mtanneau]

Is it also the case that component_lookup[k] == i if it does not equal i0, if not, then we have an issue where a previous labeling did not explore the graph correctly, right?

If component_lookup[k] != i0, then it should be equal to the index of the bus with smallest index in that connected component. I believe it should then be equal to i, yes, but only 95% confident.

[ccoffrin]

If it not the case 100% of the time then I worry the implementation has a correctness issue.

How about we add an assertion after this check that component_lookup[k] == i, this will help catch the issue down the road if we run into it.