MultiLayer Graph Attention Networks

[adrinta]

Hi @danielegrattarola,

I am wondering, how can i implement Multi-Layer GraphAttention, i try to implement it but i doubt if my code is already right or not?

• I am confused while i pass the feature matrix X_0, should i pass another adjacency matrix?
• What is the purpose of the adjacency matrix while it is replaced by attention CMIIW?

Thank you

My Multilayer GraphAttention

X_in =  Input(shape=(10,100))
A_in = Input(shape=(10,10))
X_0 = GraphAttention(16, attn_heads=4 )([X_in, A_in])
X_0 = GraphAttention(n_classes, attn_heads=4, concat_heads=False, activation='softmax')([X_0, A_in])

model = Model(inputs=[X_in, A_in], outputs=X_0)

[adrinta]

I have another question

Model1:

X_in =  Input(shape=(10,100))
A_in = Input(shape=(10,10))
X_0 = GraphAttention(16, attn_heads=4 )([X_in, A_in])
X_0 = GraphAttention(n_classes, attn_heads=4, concat_heads=False, activation='softmax')([X_0, A_in])

model = Model(inputs=[X_in, A_in], outputs=X_0)

model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              weighted_metrics=['acc'])
model.summary()

model.fit([X, A], y,
          sample_weight=train_mask,
          validation_data=validation_data,
          shuffle=False)

Model2

X_in =  Input(shape=(100,))
A_in = Input(shape=(10,))
X_0 = GraphAttention(16, attn_heads=4 )([X_in, A_in])
X_0 = GraphAttention(n_classes, attn_heads=4, concat_heads=False, activation='softmax')([X_0, A_in])

model = Model(inputs=[X_in, A_in], outputs=X_0)

model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              weighted_metrics=['acc'])
model.summary()

model.fit([X, A], y,
          sample_weight=train_mask,
          validation_data=validation_data,
          batch_size=**10**,
          shuffle=False)

what is difference Model1 and Model2, do we have to always use batch size to declare number of matrix row?
I have tried both model and get different accuracy score, Model2 is better by significance difference.

Can I get same result using Model1 with some adjustment?
i ask this because i want to produce this architecture https://ibb.co/zGHpysc, and i have problem with batch_size if i use Model2 combining the model.

The architecture is from this paper https://openaccess.thecvf.com/content_CVPR_2019/papers/Chen_Multi-Label_Image_Recognition_With_Graph_Convolutional_Networks_CVPR_2019_paper.pdf

Is possible to produce the architecture using Spektral as part of it?

Thank you very much

[danielegrattarola]

Hi,

The code to implement a multi-layer GAT is correct. You don't need to pass a different adjacency matrix because the graph structure (the edges) does not change after a GAT layer, but only the node attributes change.

The adjacency matrix is not technically replaced by the attention, but you can see it as a re-scaling. Given an edge between two nodes, the adjacency matrix will give you a "weight" of 1, while the attention mechanism will rescale that 1 to account for how important that connection is.

The two models that you posted are semantically different.
The Input layer in Keras assumes that there is an implicit batch size that you do not specify in the shape keyword.
So, your first model has inputs with shape (None, 10, 100) and (None, 10, 10), meaning that each sample given as input to the GAT is a full graph. We call this batch mode in the docs.

The second model has input shapes (None, 100) and (None, 10), which means that you have a single graph as input (see docs).

Depending on how your data is structured, one model should work and the other should crash. Also note that if you don't specify the batch size in model 1, it is automatically set to 32 by Keras.

Cheers

[laowangzi]

Hi,
I have a similar problem . The example given in citation_gat.py uses

x_in = Input(shape=(F,))
a_in = Input((N,), sparse=True)

I check the source code of SingleLoader and gat_cov.py found that it operate matrix rather than vector like shape=(F,) .So I have two problem:
1,Why the Input is not x_in = Input(shape=(N,F,))
2,Why the output of GATConv is not shape=(N,F') . But when i print the output it is a matrix of shape = (N,F')

Thank you very much! Appreciate

[danielegrattarola]

Hi,

when defining input layers, the first dimension is always None and it's implicit. In citation_gat.py, there is only one graph and therefore that None dimension is the node dimension N.
Same goes for the output, all shapes are defined as a consequence of the input.

Cheers

[laowangzi]

Many thanks for your replying