Possible to group by different size for each sample?

[lishengting]

Hi,
I have 3 samples, each is split into different number of parts, e.g. sample1 is split into 3 parts, sample2 2 parts, sample3 4 parts.
How could I group results for them?

Is it possilbe to realize it like the following way?

Channel.of( [1, 'A'], [1, 'B'], [2, 'C'], [3, 'B'], [1, 'C'], [2, 'A'], [3, 'D'] , [3, 'E'], [3, 'A']  )
    .groupTuple(size: [[1,3], [2,2], [3,4]])
    .view()

Or do you have any better suggestions?

I want specified the parts number, because when sample2 is finished, I don't want it wait for the other two samples and just continue to the next step.

Thanks a lot!

[bentsherman]

You have to use a groupKey to preserve the group size with each element:

Channel.of(
        [ 1, ['A', 'B', 'C'] ],
        [ 2, ['C', 'A'] ],
        [ 3, ['B', 'D', 'E', 'A'] ]
    )
    .map { key, values -> [ groupKey(key, values.size()), values ] }
    .transpose()
    // process the scattered elements ...
    .groupTuple()
    .map { key, values -> [ key.getGroupTarget(), values ] }
    .view()

groupTuple will recognize the groupKey and use it to emit the groups as soon as they are collected. It's also good to unwrap that groupKey afterward so that it doesn't cause any quirks down the line.

See also: groupTuple

[lishengting]

Thanks a lot for the help. I'll try it.

[xiaojie-ZENG]

Hi, I have a long pipeline, and some Channels have not been processed through groupKey.

Channel.of(
        [ 1, ['A', 'B', 'C'] ],
        [ 2, ['C', 'A'] ],
        [ 3, ['B', 'D', 'E', 'A'] ]
    )
    .map { key, values -> [ groupKey(key, values.size()), values ] }
    .transpose()
    .set{test1_input}

Channel.of([1,'sample1'],[2,'sample2'],[3,'sample3'] )
    .set{test2_input}

I found that test1_input and test2_input can be directly combined, but the channel output by test2_input after going through a process cannot be combined.

test1_input
        .combine(test2_input,by:0)
        .view()

[1, A, sample1]
[1, B, sample1]
[1, C, sample1]
[2, C, sample2]
[2, A, sample2]
[3, B, sample3]
[3, D, sample3]
[3, E, sample3]
[3, A, sample3]

process test{
    input:
        tuple val(sample),val(value1)
    output:
        tuple val(sample),val(value2), emit: test_out
    exec:
        ...
}

test3_input = test(test2_input).out.test_out
test1_input
    .combine(test3_input,by:0)
    .view()
// The above two channels cannot be combined.
 

How can I combine those two channels?

Thanks a lot.

[bentsherman]

It's probably because test1_input has the groupKey but test3_input does not. In this case what you can do is, save the group size as an extra tuple element at first, and don't use the groupKey until after you combine and before you re-group:

test1_input = Channel.of(
        [ 1, ['A', 'B', 'C'] ],
        [ 2, ['C', 'A'] ],
        [ 3, ['B', 'D', 'E', 'A'] ]
    )
    .flatMap { key, values ->
        values.collect { value ->
            [ key, values.size(), value ]
        }
    }

// later...
test3_input = test(test2_input).out.test_out
test1_input
    .combine(test3_input)
    .view()

// before re-grouping...
test3_input
    .map { key, size, value1, value3 ->
        [ groupKey(key, size), value1, value3 ]
    }
    .groupTuple()
    .view()

[xiaojie-ZENG]

It's probably because test1_input has the groupKey but test3_input does not. In this case what you can do is, save the group size as an extra tuple element at first, and don't use the groupKey until after you combine and before you re-group:

test1_input = Channel.of(
        [ 1, ['A', 'B', 'C'] ],
        [ 2, ['C', 'A'] ],
        [ 3, ['B', 'D', 'E', 'A'] ]
    )
    .flatMap { key, values ->
        values.collect { value ->
            [ key, values.size(), value ]
        }
    }

// later...
test3_input = test(test2_input).out.test_out
test1_input
    .combine(test3_input)
    .view()

// before re-grouping...
test3_input
    .map { key, size, value1, value3 ->
        [ groupKey(key, size), value1, value3 ]
    }
    .groupTuple()
    .view()

Thank you very much for your prompt reply. The method you provided indeed solves my problem.