input-order-mismatch issue in multi-input layer.

[EunjuYang]

While I'm implementing an example app. I faced an issue of the order mismatch between addLayer( ) and input tensor (multi-input layer case). Here're some descriptions on this issue.

1. I added some layers to my model, named proto_net:

  // TODO: [Bugfix] sequence of addLayer does not match with the input order.
  proto_net->addLayer(input_major_label);
  proto_net->addLayer(input_feature);
  proto_net->addLayer(l2);    /** input_feature is input of l2 **/
  proto_net->addLayer(proto); /** input_major_label and l2 are inputs of protonet **/

where the protonet consits of four layers; the last layer proto takes two input l2norm and input_major_label:

  LayerHandle proto = ml::train::createLayer(
    "protonet",
    {"name=protonet", "input_layers=[input_major_label, l2norm]",
     "maxnum_major_class=" + std::to_string(max_num_class),
     "maxnum_minor_class=" + std::to_string(max_num_class), "trainable=false"});

2. Different with my expectation, however, the actual graph's inputs do not follow the order. The input tensors' order does not matched with the addLayer order (protonet.cpp):

static constexpr size_t FIRST_INOUT_IDX = 0;
static constexpr size_t SECOND_INOUT_IDX = 1;

void Protonet::forwarding(nntrainer::RunLayerContext &context, bool training) {
  ...
  auto &features_ = context.getInput(FIRST_INOUT_IDX);
  auto &major_class_id = context.getInput(SECOND_INOUT_IDX);

This bugfix burden is passed to a user, which is required to be resolved.

(+) The order of input tensors of the layer is only affected by the order of addLayer. But it is not coherent with the order (affected but randomly decided).

(+) I tested with various combinations to boil down the isuse; I changed input_layers option in layer creation as well. The input_layers is irrelevant with the actual input tensors' order. The following code change works in the same situation:

  LayerHandle proto = ml::train::createLayer(
    "protonet",
    {"name=protonet", "input_layers=[l2norm, input_major_label]",
     "maxnum_major_class=" + std::to_string(max_num_class),
     "maxnum_minor_class=" + std::to_string(max_num_class), "trainable=false"});

---

Summary of Issue

• The order of input tensor seems to be defined by the order of addLayer when a model is created.
• However, the order is not coherent with it.
• No error or warning when the order of input_layers property does not match with the actual order.

Candidate to resolve

• It would be better to enhance our code to build the graph following the order of input_layers, not the order of addLayer.

[taos-ci]

cibot: Thank you for posting issue #2660. The person in charge will reply soon.

[lhs8928]

This issue is related with issue #1126 and #1813.

[kim-com]

Q1 : Is it correct to think that the current order follows the add_layer order?
Q2 : And should I also use the reverse order when using NeuralNetwork::inference ?

like this:

auto res = model->inference(batch, {second, first}, {});

When i create a model using configuration file(ini file), it's also have the reverse order of type = input written by configuration file(ini file)

configuration file

[model]
type = NeuralNetwork
epochs = 200
loss = mse
batch_size = 32

[optimizer]
type = adam

[in1]
type = input
input_shape = 1:2:3

[in2]
type = input
input_shape = 1:3:3

[concat]
type = concat
input_layers = in1,in2
axis = 2

[output]
type = fully_connected
input_layers = concat
unit = 1

input demension

auto input_dim = model->getInputDimension();
  for (size_t i = 0; i < input_dim.size(); i++) {
    const auto &dim = input_dim[i];
    std::cout << "INPUT_SHAPE[" << i << "] :" << dim.channel() << ", "
              << dim.height() << ", " << dim.width() << std::endl;
  }

INPUT_SHAPE[0] :1, 3, 3
INPUT_SHAPE[1] :1, 2, 3

model summarize

model->summarize(std::cout,
                     ml_train_summary_type_e::ML_TRAIN_SUMMARY_LAYER);

================================================================================
          Layer name          Layer type    Output dimension         Input layer
================================================================================
                 in2               input             1:1:3:3                    
--------------------------------------------------------------------------------
                 in1               input             1:1:2:3                    
--------------------------------------------------------------------------------
              concat              concat             1:1:5:3                 in1
                                                                             in2
--------------------------------------------------------------------------------
              output     fully_connected             1:1:5:1              concat
--------------------------------------------------------------------------------
                mse0                 mse             1:1:5:1              output
================================================================================