The dimension choose of Elmo using flair

[nininininini]

Hi, when I choose Elmo as word embedding, it seems that I can only choose a fixed dimension of 3072. Can I choose other dimensions for Elmo in flair? What part of this 3072 is composed of?

[alanakbik]

Hi @nininininini ELMo exposes three layers a 1024 states and by default we take all three to make embeddings. It is currently not possible to select only a subset of these layers, but this is a feature that could be added in a future PR.

[nininininini]

I get it, Thanks for your complete answer @alanakbik

[gabrieldevopsai]

I think that you can to use average layers.
Its possible to use flair with 1024 dims with garanted performance?

[alanakbik]

@gabrielcustodio there is a PR #1547 by @falcaopetri that adresses this issue. You can now choose whether to use a concatenation of all three layers (default) or alternatively only the topmost layer or an average. The latter two options only have 1024 dimensions!

# example sentence
sentence = Sentence("The grass is green")

# default option: concatenate all three layers (size: 3072)
embeddings = ELMoEmbeddings(embedding_mode='all')
embeddings.embed(sentence)
print(sentence[0].embedding.size())

# use only the topmost layer  (size: 1024)
embeddings = ELMoEmbeddings(embedding_mode='top')
sentence.clear_embeddings()
embeddings.embed(sentence)
print(sentence[0].embedding.size())

# use an average of all layers  (size: 1024)
embeddings = ELMoEmbeddings(embedding_mode='average')
sentence.clear_embeddings()
embeddings.embed(sentence)
print(sentence[0].embedding.size())

After the PR is merged, this feature is available in master branch and will be part of the next Flair release.