Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproduce the experiment results of TransE #14

Open
eshijia opened this issue Mar 14, 2016 · 2 comments
Open

Reproduce the experiment results of TransE #14

eshijia opened this issue Mar 14, 2016 · 2 comments

Comments

@eshijia
Copy link

eshijia commented Mar 14, 2016

With the default config in the code, I ran the FB15k_TransE.py to train a best model, and I found the evaluation result of that model was really different from what the paper said. The default epochs in the code if 500, and the paper said that the model was trained at most 1000 epochs. But current result was even much better than the result in paper. Does the paper used the code in the current repo? Below is my result.

MICRO:

-- left   >> mean: 229.41149, median: 23.0, hits@10: 37.377%
-- right  >> mean: 160.86706, median: 14.0, hits@10: 45.088%
-- global >> mean: 195.13927, median: 18.0, hits@10: 41.233%

MACRO:

-- left   >> mean: 106.30351, median: 83.18991, hits@10: 55.557%
-- right  >> mean: 84.51045, median: 63.63632, hits@10: 63.104%
-- global >> mean: 95.40698, median: 33.58689, hits@10: 59.331%
@iamaziz
Copy link

iamaziz commented Aug 19, 2016

@eshijia Not sure why, but they've mention something about reaching better results on the FB15k.

I've got exact numbers too :)

$ virtualenv --system-site-package -p /usr/local/Cellar/python/2.7.11/bin/python venv-sme
$ pip install numpy scipy
$ pip install theano 
$ source venv-sme/bin/activate
(venv-sme) $ python SME/FB15k/FB15k_TransE.py
(venv-sme) $ python SME/FB15k/FB15k_evaluation.py SME/FB15k/FB15k_TransE/best_valid_model.pkl
### MICRO:
        -- left   >> mean: 229.41149, median: 23.0, hits@10: 37.377%
        -- right  >> mean: 160.86706, median: 14.0, hits@10: 45.088%
        -- global >> mean: 195.13927, median: 18.0, hits@10: 41.233%
### MACRO:
        -- left   >> mean: 106.30351, median: 83.18991, hits@10: 55.557%
        -- right  >> mean: 84.51045, median: 63.63632, hits@10: 63.104%
        -- global >> mean: 95.40698, median: 33.58689, hits@10: 59.331%

@shanry
Copy link

shanry commented Oct 11, 2017

I got the same result. two questions: the parameters in the code is different from paper,which one is optimal? even the result of *.out file on the official github page is different frome the paper's result ,is it not funny?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants