[translation] Incorrect Download Links

[bitfort]

We have identified a problem with the download links for translation. The current download script will download the incorrect test files (the files are derived from the correct files and have additional data in them which our preprocessor does not expect). This will result in the model training correctly, but the computed BLEU scores will be incorrect.

I am currently working on a new download links and a new verify_dataset.sh.

For the time being, people can download the correct files from:

https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.en?token=ABCvMNAuy84bgb5dC-YxSxNyE6qvuit6ks5bGaqtwA%3D%3D
https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.de?token=ABCvMIGPgU8sEiCTPR4D-yovJ-TooXv2ks5bGau_wA%3D%3D

md5sum:
06e8840abe90cbfbd45cf2729807605d newstest2014.de BAD
f6c3818b477e4a25cad68b61cc883c17 newstest2014.de GOOD
4e4663b8de25d19c5fc1c4dab8d61703 newstest2014.en BAD
dabf51a9c02b2235632f3cee75c72d49 newstest2014.en GOOD

[bitfort]

To clarify, Try:

wget https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.en?token=ABCvMNAuy84bgb5dC-YxSxNyE6qvuit6ks5bGaqtwA%3D%3D -O newstest2014.en
wget https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.de?token=ABCvMIGPgU8sEiCTPR4D-yovJ-TooXv2ks5bGau_wA%3D%3D -O newstest2014.de

[vsuthichai]

Experiencing a verification issue. verify_dataset.sh correctly verifies the md5 sums for newstest2014.en and newstest2014.de but fails to verify the md5 sum for the raw_data directory.

OK: correct tensorflow/newstest2014.en
OK: correct tensorflow/newstest2014.de
ERROR: incorrect raw_data/
ERROR: expected 8fd41a5c658948dfbc1ec83751d9c7fe  -
ERROR: found b21fdd71ac1056a4be466617501bdecf  -

Here is the output of find raw_data/ -type f -exec md5sum {} \; | sort -k 2

c8685312058e23cf08f46abe444a332f  raw_data/commoncrawl.cs-en.annotation
9099f254791097f42952e10f20087cf5  raw_data/commoncrawl.cs-en.cs
480e7d6c2a040ce13fe835a7a0f0017a  raw_data/commoncrawl.cs-en.en
37b003a69178ec22cb3d74a8d9bfb7cc  raw_data/commoncrawl.de-en.annotation
ac590e7b7a1d04ea67b80a6056738dbc  raw_data/commoncrawl.de-en.de
e3d852042ff03ea0ea98d828b98b6c2b  raw_data/commoncrawl.de-en.en
edfa5b076cc2612ddcba84e78096667a  raw_data/commoncrawl.es-en.annotation
34e50983aee1342fcc6ce3cdca085876  raw_data/commoncrawl.es-en.en
f413a156a8a77ffa9ee9d15db338dd12  raw_data/commoncrawl.es-en.es
ed322913456bd3c8288eefc8658c0039  raw_data/commoncrawl.fr-en.annotation
6ac43d37120ebd9815b8b020fc9c479d  raw_data/commoncrawl.fr-en.en
e2c8b6d4ef2c70eb4d0db0c0fc4cc764  raw_data/commoncrawl.fr-en.fr
9598aaa8d74ec7f6265182501e40d05c  raw_data/commoncrawl.ru-en.annotation
4d7b9868a9f0e47c55161b809ed67071  raw_data/commoncrawl.ru-en.en
2af57bbc46a35996c35a728704bb73ce  raw_data/commoncrawl.ru-en.ru
9dc5dd380525793b8e143a3f0dd25ce7  raw_data/dev/newsdev2014.en
f410731105f7cd313148069f096e34c3  raw_data/dev/newsdev2014.hi
042471a75ba56278776ca74e54d167f2  raw_data/dev/newsdev2014-ref.en.sgm
ec3458c0e785dee77fadcd9904fe3941  raw_data/dev/.newsdev2014-ref.en.sgm.swp
0e4a581b13f190c280e4f05985813482  raw_data/dev/newsdev2014-ref.hi.sgm
4c5248f9c7462945c38e857aeb1d6170  raw_data/dev/newsdev2014-src.en.sgm
04144c2031834adcfaf769ce4f18eaaa  raw_data/dev/newsdev2014-src.hi.sgm
b8d952eff69c69ddcde8ad4d0fcd3d57  raw_data/dev/newsdev2015-enfi-ref.fi.sgm
6f04d7c577710c309630f5e95e46eab5  raw_data/dev/newsdev2015-enfi-src.en.sgm
af8109c2ddd33ee2a9f55a21acc4c739  raw_data/dev/newsdev2015-fien-ref.en.sgm
775365f4fe63a7fc09324bfc66bf0428  raw_data/dev/newsdev2015-fien-src.fi.sgm
bc008c5abd9460272633133f141eaf2e  raw_data/dev/newsdev2016-enro-ref.ro.sgm
ed83751e0e925f26a72a7874664be4c0  raw_data/dev/newsdev2016-enro-src.en.sgm
3022225d15ac3688b8166aa8dd236413  raw_data/dev/newsdev2016-entr-ref.tr.sgm
03af6a305a2b90b97364a6032c2b4242  raw_data/dev/newsdev2016-entr-src.en.sgm
1b5b8dedd023369daad6d384048df653  raw_data/dev/newsdev2016-roen-ref.en.sgm
5405823055b9bf13174bbddbe1fd9a41  raw_data/dev/newsdev2016-roen-src.ro.sgm
2685357bce4cae487245e336a832af2d  raw_data/dev/newsdev2016-tren-ref.en.sgm
d0ce333bc57400b03a86e525fe1f5a85  raw_data/dev/newsdev2016-tren-src.tr.sgm
5d7431cfdc0db5a5442c063b6fc96108  raw_data/dev/newsdev2017-enlv-ref.lv.sgm
bb11d739392638db0a35aa376718cda1  raw_data/dev/newsdev2017-enlv-src.en.sgm
e8eee5ded310f6b4cf1c8056a472b455  raw_data/dev/newsdev2017-enzh-ref.zh.sgm
22a0f21e8ddfec362c1fb15adcb957d4  raw_data/dev/newsdev2017-enzh-src.en.sgm
1df951c6a6cf67f45668eff6b863c086  raw_data/dev/newsdev2017-lven-ref.en.sgm
4b422caddea73dad2dd345083bd7c18e  raw_data/dev/newsdev2017-lven-src.lv.sgm
f6e050792e388cb3215e193ca1c8c7c1  raw_data/dev/newsdev2017-zhen-ref.en.sgm
c226046ad0365e3f5b593a5f08b6465a  raw_data/dev/newsdev2017-zhen-src.zh.sgm
a946a6b7b0fc5f63970d7ba2b8196c7b  raw_data/dev/newsdiscussdev2015-enfr-ref.fr.sgm
cd1b574185d964ddbe58d6f4b6a30b95  raw_data/dev/newsdiscussdev2015-enfr-src.en.sgm
05cb1f18c1f811e500c62dde2ed89da4  raw_data/dev/newsdiscussdev2015-fren-ref.en.sgm
90a841d07167f718dd42a9f324eeb940  raw_data/dev/newsdiscussdev2015-fren-src.fr.sgm
51958eb354f2d1dd7621b3988b52359f  raw_data/dev/newsdiscusstest2015-enfr-ref.fr.sgm
05bf9e40ae27a8e9a6c40370ca0fdb53  raw_data/dev/newsdiscusstest2015-enfr-src.en.sgm
d3492954af94acae57871af3dd46dcfe  raw_data/dev/newsdiscusstest2015-fren-ref.en.sgm
18cc8a22b83b6fec68fe3283e8a2ef3e  raw_data/dev/newsdiscusstest2015-fren-src.fr.sgm
ca72f09204b5a1e7c5b046774394a87e  raw_data/dev/newssyscomb2009.cs
1b15e5b748e1fd085dbedf67869c87ef  raw_data/dev/newssyscomb2009.de
a0efdc23a268bcc858c528856d85b33a  raw_data/dev/newssyscomb2009.en
f238c8a58aa6d8bf65088287664457aa  raw_data/dev/newssyscomb2009.es
27dedff510afd58ed76e66e9a725070f  raw_data/dev/newssyscomb2009.fr
cf705039c93cda9a9f433fffe376cb34  raw_data/dev/newssyscomb2009-ref.cs.sgm
063ac2b01f9639e8557870cee9653c7f  raw_data/dev/newssyscomb2009-ref.de.sgm
78e1ae9398be9999c431c436005a1670  raw_data/dev/newssyscomb2009-ref.en.sgm
0d2bca154e433e225613cf5bcc777bff  raw_data/dev/newssyscomb2009-ref.es.sgm
ef320296f23773b214dac3b49036c561  raw_data/dev/newssyscomb2009-ref.fr.sgm
e24724599b481ee4231f0681964c7a6b  raw_data/dev/newssyscomb2009-ref.hu.sgm
e71d224b851fb77014e22534cfa26a70  raw_data/dev/newssyscomb2009-ref.it.sgm
afbb687ec4b1584f0a85c9092133a161  raw_data/dev/newssyscomb2009-src.cs.sgm
b11fa2f16d70abc5dcfe90b8d0167e61  raw_data/dev/newssyscomb2009-src.de.sgm
2126a71c2c726aec3302418533ab6e7f  raw_data/dev/newssyscomb2009-src.en.sgm
3632ca861af5ab02be27aa6436a5a44f  raw_data/dev/newssyscomb2009-src.es.sgm
1a52f5ecef88ab4f834027bcf808770e  raw_data/dev/newssyscomb2009-src.fr.sgm
7d99d4c4284be0aec304649e831bc49d  raw_data/dev/newssyscomb2009-src.hu.sgm
97f5a343952b3370b7c9569ad8035fac  raw_data/dev/newssyscomb2009-src.it.sgm
04b629a9593caa55cb91f60b8ff1e65c  raw_data/dev/news-test2008.cs
f9d7757d029d05d513e81c4ba42d3e03  raw_data/dev/news-test2008.de
49669d8139206c44b6d5daaedf96f219  raw_data/dev/news-test2008.en
8e39422f7e6eed2862e7c442fe279e0d  raw_data/dev/news-test2008.es
d4bb111afbd15564b0e9611783bbee8e  raw_data/dev/news-test2008.fr
73d58db5f47e4e2c5ca1af275ad8805d  raw_data/dev/news-test2008-ref.cs.sgm
8c8f84de9ac3ff60553433b483562fb3  raw_data/dev/news-test2008-ref.de.sgm
ab6551704a20689039db8472b88cc5ea  raw_data/dev/news-test2008-ref.en.sgm
f1c9b0ed953233ff3a96dca014258748  raw_data/dev/news-test2008-ref.es.sgm
02a31f4e26a1b7a4987998f08eb13aad  raw_data/dev/news-test2008-ref.fr.sgm
8e5d54cd743a0d430c7366c86fd55d6f  raw_data/dev/news-test2008-ref.hu.sgm
a2d904389c4bc3a43bce92f7eea9a0e9  raw_data/dev/news-test2008-src.cs.sgm
cb85b28178e8ae6ff1ece09c0628432b  raw_data/dev/news-test2008-src.de.sgm
64e5fb7994b49390aa11f34b222dc870  raw_data/dev/news-test2008-src.en.sgm
2ade1a853d72bc39c12efae03c5a9dd1  raw_data/dev/news-test2008-src.es.sgm
6dbcedd2d42114fac5fc9194c406403b  raw_data/dev/news-test2008-src.fr.sgm
4d83b15d675e0eb85a91b12d9fba856f  raw_data/dev/news-test2008-src.hu.sgm
61a59053e968b66d1c7aa0acb8edd8bb  raw_data/dev/newstest2009.cs
2c30a44320eed1ab4a882efd02857bdd  raw_data/dev/newstest2009.de
ef5a07243f3081d0d5ecd3524b82a4b3  raw_data/dev/newstest2009.en
3067bda4778b2adfa9cab2bc012208fb  raw_data/dev/newstest2009.es
44521b48ec378ab93ee9bde2d4891100  raw_data/dev/newstest2009.fr
0e509a8b7c7f3488fb795ca28122fd5e  raw_data/dev/newstest2009-ref.cs.sgm
1ac2d0977f3f66f77b0ff6eba4204f8a  raw_data/dev/newstest2009-ref.de.sgm
02752b423282def0fde39d8642881474  raw_data/dev/newstest2009-ref.en.sgm
20c09089345db76d1544ad47c6fa7a83  raw_data/dev/newstest2009-ref.es.sgm
17ff6e37c457594cf8692696d428aa8b  raw_data/dev/newstest2009-ref.fr.sgm
44f295f05985a748141ec594e4150961  raw_data/dev/newstest2009-ref.hu.sgm
2d8d509b81fb7391898042ce0d98e563  raw_data/dev/newstest2009-ref.it.sgm
8eac2eff3480140fc1a4ad410dce1900  raw_data/dev/newstest2009-src.cs.sgm
18ab956bd1bbfa6b51b462ddfc06662c  raw_data/dev/newstest2009-src.de.sgm
d62ac3e6acd9c5d3567328e92013953b  raw_data/dev/newstest2009-src.en.sgm
e73d24ad2817e7638102ef596032aecb  raw_data/dev/newstest2009-src.es.sgm
c4928fd5607d05ac7f9ce4955ccba5e2  raw_data/dev/newstest2009-src.fr.sgm
27b66908bf26e7e4208e5e1b0b53f99f  raw_data/dev/newstest2009-src.hu.sgm
ec688093aa4df5e9e2625f2cd8f1e72c  raw_data/dev/newstest2009-src.it.sgm
d62ac3e6acd9c5d3567328e92013953b  raw_data/dev/newstest2009-src.xx.sgm
44fe0c72ac6008c644191e2dee8915d3  raw_data/dev/newstest2010.cs
aa3d59046db19ddf9997697c68c3c74b  raw_data/dev/newstest2010.de
9d23aa9d0be838be2f85a7bff78ce6f0  raw_data/dev/newstest2010.en
0014ee0b25fe7ca304a7e62ff18e3881  raw_data/dev/newstest2010.es
6136fb2113ddfa48039ea7b261ee9c46  raw_data/dev/newstest2010.fr
85449fc49c7c9a8729ab0820c8bdfe21  raw_data/dev/newstest2010-ref.cs.sgm
dcf4690dfaca3e33537fcef4cbddd9fd  raw_data/dev/newstest2010-ref.de.sgm
2052be37e7902b5d700ca1c196f44a11  raw_data/dev/newstest2010-ref.en.sgm
1084df7ad0ac842f1f30afdd34865683  raw_data/dev/newstest2010-ref.es.sgm
eb8a41f2e16e0886975b477d6af2a92a  raw_data/dev/newstest2010-ref.fr.sgm
57f4e6ce5281e501dd20cc563f2df4df  raw_data/dev/newstest2010-src.cs.sgm
f4e25f698ef4f304eba8daa94fe46bfd  raw_data/dev/newstest2010-src.de.sgm
032fab9462c87548e36e8a8aeca41744  raw_data/dev/newstest2010-src.en.sgm
2a08ef80f9fbc61ffaf58b91dc08224c  raw_data/dev/newstest2010-src.es.sgm
97cf227b62ba94003ccd7c39236ea79e  raw_data/dev/newstest2010-src.fr.sgm
32051fcf0eb1cad6b5351523af90893c  raw_data/dev/newstest2011.cs
c09c3e1883b2cd081f50021b69887ce9  raw_data/dev/newstest2011.de
42244c28949590deb74b593f37d6a82c  raw_data/dev/newstest2011.en
994033e6b8546c7b4810a36cfc49cbd5  raw_data/dev/newstest2011.es
60772053ba2332b99db7a5bb84ad6c47  raw_data/dev/newstest2011.fr
da81b878fa01de717c2e38dfe74d0662  raw_data/dev/newstest2011-ref.cs.sgm
206b22bf0d2f286bd13e87925ee53c16  raw_data/dev/newstest2011-ref.de.sgm
bafdf8e5de7c243b9d937f405b1e208c  raw_data/dev/newstest2011-ref.en.sgm
4b86af5befd5eb1ad3f0b8e51e9a4693  raw_data/dev/newstest2011-ref.es.sgm
72a65f0be1f34a843fa646fcd085e085  raw_data/dev/newstest2011-ref.fr.sgm
74385262f92b7936ec760a2f70b5c2e1  raw_data/dev/newstest2011-src.cs.sgm
c768d2d7c1d59f756b19b0b282458659  raw_data/dev/newstest2011-src.de.sgm
15f66b409b0fbd51e3d266042d57dc47  raw_data/dev/newstest2011-src.en.sgm
8a308fededd132bfc6eb34b7d4d8238d  raw_data/dev/newstest2011-src.es.sgm
768beed6d8a4ba3693ed2593ee155efb  raw_data/dev/newstest2011-src.fr.sgm
03ce56fa8170e61602aaa6fc38c4f2b8  raw_data/dev/newstest2012.cs
14823046a5a438a1d1c2f09d22f38baa  raw_data/dev/newstest2012.de
21b17e5a2dd710b6fc6d659d4498cbb4  raw_data/dev/newstest2012.en
7f56040187fc9d4f6dbe927465cbaa2b  raw_data/dev/newstest2012.es
12922ad9ffe8c65d3bf1f8076aea19ad  raw_data/dev/newstest2012.fr
48a2e70a7882e5977668be7b5ff16122  raw_data/dev/newstest2012-ref.cs.sgm
77debbee91095e88466a2fcbc01ecb10  raw_data/dev/newstest2012-ref.de.sgm
4d5efd442bb2f843523deca1a36aafd1  raw_data/dev/newstest2012-ref.en.sgm
b5b41d0b6ba15dc79810e07647dd58af  raw_data/dev/newstest2012-ref.es.sgm
c9e4b546b91414b34196def73e01594e  raw_data/dev/newstest2012-ref.fr.sgm
0792ba330cb2aaa6c7060618df309613  raw_data/dev/newstest2012-ref.ru.sgm
2dad778e5008d6f0e8607b35ae5f05bc  raw_data/dev/newstest2012.ru
649fcfbd8d8b706bd7f75e30d775d647  raw_data/dev/newstest2012-src.cs.sgm
91b8024827c7a18cd3c4d8d5805c6844  raw_data/dev/newstest2012-src.de.sgm
1af3633cab6c85e71afe46d235f8a841  raw_data/dev/newstest2012-src.en.sgm
0de786bf7a22c7ff32db808f2375d26a  raw_data/dev/newstest2012-src.es.sgm
026b732bb653053bbc02fdb6244e0662  raw_data/dev/newstest2012-src.fr.sgm
ce8910c192df6c3d52632cd3faf5bcd3  raw_data/dev/newstest2012-src.ru.sgm
6a7e7af5a91b49bffcd7749312c029b4  raw_data/dev/newstest2013.cs
872891e01855b43f2a87155e0c1d2076  raw_data/dev/newstest2013.de
23be23d7b9bd5ad1f2ebd076484332c9  raw_data/dev/newstest2013.en
2bc840a2c850d883532d66c67b4f25ee  raw_data/dev/newstest2013.es
77db6a3d2259d00b4d5f45009a3ed375  raw_data/dev/newstest2013.fr
f4d4847b15bd85cfc0578850ebd88edb  raw_data/dev/newstest2013-ref.cs.sgm
87bb842dc694d65c703d75eaa2ec49e4  raw_data/dev/newstest2013-ref.de.sgm
c6d6bd214bc3de533b619011d6f7f8ce  raw_data/dev/newstest2013-ref.en.sgm
a72c9746b673fe4bb43caca6f6c42fa2  raw_data/dev/.newstest2013-ref.en.sgm.swp
e31fd14c8f4b8035376093d1d6126fa9  raw_data/dev/newstest2013-ref.es.sgm
d66bb0403b4e1cb7fb7eb5751206176c  raw_data/dev/newstest2013-ref.fr.sgm
579c0d03ea715d4c2c633a561cdf67fc  raw_data/dev/newstest2013-ref.ru.sgm
1aed45941a2a4fca5d7c54d4ca63f67b  raw_data/dev/newstest2013.ru
afe7f6893521e199391420ff725ed23d  raw_data/dev/newstest2013-src.cs.sgm
4c417abe9ac3183a804f29c285641506  raw_data/dev/newstest2013-src.de.sgm
b86216894fdf76faced26c301143627e  raw_data/dev/newstest2013-src.en.sgm
dff4e9a5696833b338b1356c4c0f7f06  raw_data/dev/newstest2013-src.es.sgm
c6b79404ee5f581e579abf654622ad5f  raw_data/dev/newstest2013-src.fr.sgm
78cf164d8e24d55e31aea44b550be3d4  raw_data/dev/newstest2013-src.ru.sgm
0f6923bec65877e6395f73cc2565b596  raw_data/dev/newstest2014-csen-ref.cs.sgm
cf965a55c0f46b6822b424b21af4eb2f  raw_data/dev/newstest2014-csen-ref.en.sgm
2ece2d0b521e33618afff968a49d4d4b  raw_data/dev/newstest2014-csen-src.cs.sgm
393cffbc087377c98035ccd73c8a8a7a  raw_data/dev/newstest2014-csen-src.en.sgm
a05cee5504baa520c1a63b104bb56c87  raw_data/dev/newstest2014-deen-ref.de.sgm
7f907030087b88aeae0303f11378afa4  raw_data/dev/newstest2014-deen-ref.en.sgm
7183d7875ea7ba4a12731140acc8cc8f  raw_data/dev/newstest2014-deen-src.de.sgm
5bc31d8d48424640197b89e00cedabb2  raw_data/dev/newstest2014-deen-src.en.sgm
b1dfacd377a34f4f9927dd0947783a1e  raw_data/dev/newstest2014-fren-ref.en.sgm
4101e1a3ff6c5ad23aa2f6e0ce0f22bb  raw_data/dev/newstest2014-fren-ref.fr.sgm
7b92b0c92b4b7a077bf4cc055142231d  raw_data/dev/newstest2014-fren-src.en.sgm
cfd865f26bae037d2db50514d668b195  raw_data/dev/newstest2014-fren-src.fr.sgm
bbe5c5d92bae574967ebcd3af0e658ba  raw_data/dev/newstest2014-hien-ref.en.sgm
d04f76b6bf57ec79970cf0f54906573d  raw_data/dev/newstest2014-hien-ref.hi.sgm
10ca55c71784fb4ef8f1c612a8e336f0  raw_data/dev/newstest2014-hien-src.en.sgm
1a5f7438ad30f80942f07085225c7dfe  raw_data/dev/newstest2014-hien-src.hi.sgm
f97b79ab2a9723be1afcbcadbabb491e  raw_data/dev/newstest2014-ruen-ref.en.sgm
4ea80825f9fe641a4327e4c492d253c1  raw_data/dev/newstest2014-ruen-ref.ru.sgm
82f5d4f204485c72203512c9d814170a  raw_data/dev/newstest2014-ruen-src.en.sgm
1f22dccd51da1f9163508213a183b45e  raw_data/dev/newstest2014-ruen-src.ru.sgm
e7d924c9ca7baa23baae68e2cee7cd54  raw_data/dev/newstest2015-csen-ref.en.sgm
9a29cac72da00f718f88eda30c4ac278  raw_data/dev/newstest2015-csen-src.cs.sgm
8fb26a529c52fbabab17222e97b612d1  raw_data/dev/newstest2015-deen-ref.en.sgm
0dfca16a9c9ad3aea12fdb5c1085efab  raw_data/dev/newstest2015-deen-src.de.sgm
0238bdf703da9d81d75f5cba3bcecd0d  raw_data/dev/newstest2015-encs-ref.cs.sgm
aa067ef9b7fac031e18d6df357e576d5  raw_data/dev/newstest2015-encs-src.en.sgm
d0cf8dfcd376b2c9136eaba9944bd1a8  raw_data/dev/newstest2015-ende-ref.de.sgm
ade0000673b5a2f0229ec34b3befa950  raw_data/dev/newstest2015-ende-src.en.sgm
bd5b5bbeee6e58ecc2477a3cadebb10b  raw_data/dev/newstest2015-enfi-ref.fi.sgm
7e3e983a139c570b776dd0734dfb40cf  raw_data/dev/newstest2015-enfi-src.en.sgm
9048de63ac7ceb0ba7ba000babc0cde9  raw_data/dev/newstest2015-enru-ref.ru.sgm
a49f5622aba3461351958a55582da029  raw_data/dev/newstest2015-enru-src.en.sgm
92ad86105ecf7faed99ed97a99b13c9f  raw_data/dev/newstest2015-fien-ref.en.sgm
d6526c035382f5d0e4a0c967c1572ab9  raw_data/dev/newstest2015-fien-src.fi.sgm
71428fcc33f147d7a7ee51beb49d6d6a  raw_data/dev/newstest2015-ruen-ref.en.sgm
e99f900c5cdaacb38e6a8bd55e8030d4  raw_data/dev/newstest2015-ruen-src.ru.sgm
07d4ee9bb1af67805610ab4731c2aafa  raw_data/dev/newstest2016-csen-ref.en.sgm
8f20636841df645b7bb400f7929f5c46  raw_data/dev/newstest2016-csen-src.cs.sgm
3a34a87b0305e3813801fbc097f1dd7d  raw_data/dev/newstest2016-deen-ref.en.sgm
23d3fc0b52b863d8ac850d16de2d4794  raw_data/dev/newstest2016-deen-src.de.sgm
092a21aae1a69118d9a0799159f8b72f  raw_data/dev/newstest2016-encs-ref.cs.sgm
9fb44e0a85f30bcdc6f779532538643e  raw_data/dev/newstest2016-encs-src.en.sgm
381626ee8249023895c30610bbce456b  raw_data/dev/newstest2016-ende-ref.de.sgm
2677196bd9afddd6b01b4a6785ec67e1  raw_data/dev/newstest2016-ende-src.en.sgm
aa340b116fa0fa3aa83af2b519796b74  raw_data/dev/newstest2016-enfi-ref.fi.sgm
742fb7aa120742aaf1aef9b1f59356d2  raw_data/dev/newstest2016-enfi-src.en.sgm
e0498e26140a0c695515af473fcf72e0  raw_data/dev/newstest2016-enro-ref.ro.sgm
77f598ea5d0b97c053e6b3130f19a2a1  raw_data/dev/newstest2016-enro-src.en.sgm
8efdb854c94d7d87587a65f93be66ce8  raw_data/dev/newstest2016-enru-ref.ru.sgm
ff756d95230800aa7d06f56dcf08a3dc  raw_data/dev/newstest2016-enru-src.en.sgm
2a74a2919942b5ddca4e1e7743aea147  raw_data/dev/newstest2016-entr-ref.tr.sgm
a966abbbeb554b47429cdff61f3d56d5  raw_data/dev/newstest2016-entr-src.en.sgm
8cdb386191e814ae3cc0215da6822747  raw_data/dev/newstest2016-fien-ref.en.sgm
b061d885ee902680edafe421afa16e9b  raw_data/dev/newstest2016-fien-src.fi.sgm
8ebbda7d07414c791d4c0bb5897f5adc  raw_data/dev/newstest2016-roen-ref.en.sgm
59593b5ce866ae5483e6eaadf762033d  raw_data/dev/newstest2016-roen-src.ro.sgm
1e0163031a77e40b6fec8b7dd8288fc0  raw_data/dev/newstest2016-ruen-ref.en.sgm
55e2ea602accd822369434eac47625ba  raw_data/dev/newstest2016-ruen-src.ru.sgm
ab9942174352e2498a522865cb536db2  raw_data/dev/newstest2016-tren-ref.en.sgm
660635a258aac67df902d4f7d4865f0e  raw_data/dev/newstest2016-tren-src.tr.sgm
ff85f980e33deddca6902bb8034673e4  raw_data/dev/newstestB2016-enfi-ref.fi.sgm
66993925bb1dac7915a93066892f6bb4  raw_data/dev/newstestB2016-enfi-src.en.sgm
9b1aa63c1cf49dccdd20b962fe313989  raw_data/dev.tgz
b4ea563c1b08f6f01689d3cb9948be54  raw_data/training/europarl-v7.cs-en.cs
ce2d7fa76e871288ffa2af5985424b43  raw_data/training/europarl-v7.cs-en.en
f77dd3ab69ea7cc7ef9248e4d96449e1  raw_data/training/europarl-v7.de-en.de
56a330fd9a6291c2540ee733e40e9dd2  raw_data/training/europarl-v7.de-en.en
df1dbeda8edd3838d475dfab242595b3  raw_data/training/europarl-v7.es-en.en
17d03c84750988fdf84705ef4935db7f  raw_data/training/europarl-v7.es-en.es
ee3e98f101300baccd2b46bb2592b66e  raw_data/training/europarl-v7.fr-en.en
fb585c23a9b38555b8a3f1141d73c5f2  raw_data/training/europarl-v7.fr-en.fr
80bd3d7ab52431574dcf42b8cbc0662c  raw_data/training/news-commentary-v12.cs-en.cs
e794ce48b4dfa1f467714a18f99e6bc9  raw_data/training/news-commentary-v12.cs-en.en
2af641b908b038ef198f7e42503c4487  raw_data/training/news-commentary-v12.de-en.de
3ccbc53e3d65efe8db528e7550d00e98  raw_data/training/news-commentary-v12.de-en.en
e08f590c0687fbc2c6d6ab3f80aa670f  raw_data/training/news-commentary-v12.es-en.en
b815e8154b7ac3cc7314eb7bd72c6415  raw_data/training/news-commentary-v12.es-en.es
3a4403b8de5822e69cb00a02674feb11  raw_data/training/news-commentary-v12.fr-en.en
65e171cd3810aaf05ee9ce22cf729c50  raw_data/training/news-commentary-v12.fr-en.fr
8c2761ddf974ed1fc1ef26cc83d3999d  raw_data/training/news-commentary-v12.ru-en.en
d727222cee01d49e5dc7c5acd223f4c6  raw_data/training/news-commentary-v12.ru-en.ru
09270cbb14bcdfc57d2e9dbc678418f1  raw_data/training/news-commentary-v12.zh-en.en
91ad2ff8e9b9a9c7eccd4faec85b239e  raw_data/training/news-commentary-v12.zh-en.zh
7e0acbe86b0d7816300e14650f5b2bd4  raw_data/training-parallel-commoncrawl.tgz
c52404583294a1b609e56d45b2ed06f5  raw_data/training-parallel-europarl-v7.tgz
fc6b83b809347e64f511d291e4bc8731  raw_data/training-parallel-nc-v12.tgz

[bitfort]

I'm taking a look at this... will get back to you soon.

[xyhuang]

Looks like the differences are the following files:

23be23d7b9bd5ad1f2ebd076484332c9  raw_data/wmt32k-dev.lang1
872891e01855b43f2a87155e0c1d2076  raw_data/wmt32k-dev.lang2
1479de83f154a2f3393f407a54a98be6  raw_data/wmt32k-train.lang1
f2be901a1248a740265273dd971819c0  raw_data/wmt32k-train.lang2

These are actually generated by preprocessing. Maybe the checksum should be changed to exclude these files.

[jbalma]

After running ./download_data.sh into raw_data/ I run the verify_dataset.sh script after it completes. I'm hitting a similar issue with using the verify_dataset.sh as the users above with the MD5 sum failing with

OK: correct tensorflow/newstest2014.en
OK: correct tensorflow/newstest2014.de
ERROR: incorrect raw_data/
ERROR: expected 8fd41a5c658948dfbc1ec83751d9c7fe -
ERROR: found fb4469fa118fdae84f948f65c2a75ac8 -

Here's the output of find raw_data/ -type f -exec md5sum {} ; | sort -k 2

c8685312058e23cf08f46abe444a332f raw_data/commoncrawl.cs-en.annotation
9099f254791097f42952e10f20087cf5 raw_data/commoncrawl.cs-en.cs
480e7d6c2a040ce13fe835a7a0f0017a raw_data/commoncrawl.cs-en.en
37b003a69178ec22cb3d74a8d9bfb7cc raw_data/commoncrawl.de-en.annotation
ac590e7b7a1d04ea67b80a6056738dbc raw_data/commoncrawl.de-en.de
e3d852042ff03ea0ea98d828b98b6c2b raw_data/commoncrawl.de-en.en
edfa5b076cc2612ddcba84e78096667a raw_data/commoncrawl.es-en.annotation
34e50983aee1342fcc6ce3cdca085876 raw_data/commoncrawl.es-en.en
f413a156a8a77ffa9ee9d15db338dd12 raw_data/commoncrawl.es-en.es
ed322913456bd3c8288eefc8658c0039 raw_data/commoncrawl.fr-en.annotation
6ac43d37120ebd9815b8b020fc9c479d raw_data/commoncrawl.fr-en.en
e2c8b6d4ef2c70eb4d0db0c0fc4cc764 raw_data/commoncrawl.fr-en.fr
9598aaa8d74ec7f6265182501e40d05c raw_data/commoncrawl.ru-en.annotation
4d7b9868a9f0e47c55161b809ed67071 raw_data/commoncrawl.ru-en.en
2af57bbc46a35996c35a728704bb73ce raw_data/commoncrawl.ru-en.ru
9b1aa63c1cf49dccdd20b962fe313989 raw_data/dev.tgz
ec3458c0e785dee77fadcd9904fe3941 raw_data/dev/.newsdev2014-ref.en.sgm.swp
a72c9746b673fe4bb43caca6f6c42fa2 raw_data/dev/.newstest2013-ref.en.sgm.swp
73d58db5f47e4e2c5ca1af275ad8805d raw_data/dev/news-test2008-ref.cs.sgm
8c8f84de9ac3ff60553433b483562fb3 raw_data/dev/news-test2008-ref.de.sgm
ab6551704a20689039db8472b88cc5ea raw_data/dev/news-test2008-ref.en.sgm
f1c9b0ed953233ff3a96dca014258748 raw_data/dev/news-test2008-ref.es.sgm
02a31f4e26a1b7a4987998f08eb13aad raw_data/dev/news-test2008-ref.fr.sgm
8e5d54cd743a0d430c7366c86fd55d6f raw_data/dev/news-test2008-ref.hu.sgm
a2d904389c4bc3a43bce92f7eea9a0e9 raw_data/dev/news-test2008-src.cs.sgm
cb85b28178e8ae6ff1ece09c0628432b raw_data/dev/news-test2008-src.de.sgm
64e5fb7994b49390aa11f34b222dc870 raw_data/dev/news-test2008-src.en.sgm
2ade1a853d72bc39c12efae03c5a9dd1 raw_data/dev/news-test2008-src.es.sgm
6dbcedd2d42114fac5fc9194c406403b raw_data/dev/news-test2008-src.fr.sgm
4d83b15d675e0eb85a91b12d9fba856f raw_data/dev/news-test2008-src.hu.sgm
04b629a9593caa55cb91f60b8ff1e65c raw_data/dev/news-test2008.cs
f9d7757d029d05d513e81c4ba42d3e03 raw_data/dev/news-test2008.de
49669d8139206c44b6d5daaedf96f219 raw_data/dev/news-test2008.en
8e39422f7e6eed2862e7c442fe279e0d raw_data/dev/news-test2008.es
d4bb111afbd15564b0e9611783bbee8e raw_data/dev/news-test2008.fr
042471a75ba56278776ca74e54d167f2 raw_data/dev/newsdev2014-ref.en.sgm
0e4a581b13f190c280e4f05985813482 raw_data/dev/newsdev2014-ref.hi.sgm
4c5248f9c7462945c38e857aeb1d6170 raw_data/dev/newsdev2014-src.en.sgm
04144c2031834adcfaf769ce4f18eaaa raw_data/dev/newsdev2014-src.hi.sgm
9dc5dd380525793b8e143a3f0dd25ce7 raw_data/dev/newsdev2014.en
f410731105f7cd313148069f096e34c3 raw_data/dev/newsdev2014.hi
b8d952eff69c69ddcde8ad4d0fcd3d57 raw_data/dev/newsdev2015-enfi-ref.fi.sgm
6f04d7c577710c309630f5e95e46eab5 raw_data/dev/newsdev2015-enfi-src.en.sgm
af8109c2ddd33ee2a9f55a21acc4c739 raw_data/dev/newsdev2015-fien-ref.en.sgm
775365f4fe63a7fc09324bfc66bf0428 raw_data/dev/newsdev2015-fien-src.fi.sgm
bc008c5abd9460272633133f141eaf2e raw_data/dev/newsdev2016-enro-ref.ro.sgm
ed83751e0e925f26a72a7874664be4c0 raw_data/dev/newsdev2016-enro-src.en.sgm
3022225d15ac3688b8166aa8dd236413 raw_data/dev/newsdev2016-entr-ref.tr.sgm
03af6a305a2b90b97364a6032c2b4242 raw_data/dev/newsdev2016-entr-src.en.sgm
1b5b8dedd023369daad6d384048df653 raw_data/dev/newsdev2016-roen-ref.en.sgm
5405823055b9bf13174bbddbe1fd9a41 raw_data/dev/newsdev2016-roen-src.ro.sgm
2685357bce4cae487245e336a832af2d raw_data/dev/newsdev2016-tren-ref.en.sgm
d0ce333bc57400b03a86e525fe1f5a85 raw_data/dev/newsdev2016-tren-src.tr.sgm
5d7431cfdc0db5a5442c063b6fc96108 raw_data/dev/newsdev2017-enlv-ref.lv.sgm
bb11d739392638db0a35aa376718cda1 raw_data/dev/newsdev2017-enlv-src.en.sgm
e8eee5ded310f6b4cf1c8056a472b455 raw_data/dev/newsdev2017-enzh-ref.zh.sgm
22a0f21e8ddfec362c1fb15adcb957d4 raw_data/dev/newsdev2017-enzh-src.en.sgm
1df951c6a6cf67f45668eff6b863c086 raw_data/dev/newsdev2017-lven-ref.en.sgm
4b422caddea73dad2dd345083bd7c18e raw_data/dev/newsdev2017-lven-src.lv.sgm
f6e050792e388cb3215e193ca1c8c7c1 raw_data/dev/newsdev2017-zhen-ref.en.sgm
c226046ad0365e3f5b593a5f08b6465a raw_data/dev/newsdev2017-zhen-src.zh.sgm
a946a6b7b0fc5f63970d7ba2b8196c7b raw_data/dev/newsdiscussdev2015-enfr-ref.fr.sgm
cd1b574185d964ddbe58d6f4b6a30b95 raw_data/dev/newsdiscussdev2015-enfr-src.en.sgm
05cb1f18c1f811e500c62dde2ed89da4 raw_data/dev/newsdiscussdev2015-fren-ref.en.sgm
90a841d07167f718dd42a9f324eeb940 raw_data/dev/newsdiscussdev2015-fren-src.fr.sgm
51958eb354f2d1dd7621b3988b52359f raw_data/dev/newsdiscusstest2015-enfr-ref.fr.sgm
05bf9e40ae27a8e9a6c40370ca0fdb53 raw_data/dev/newsdiscusstest2015-enfr-src.en.sgm
d3492954af94acae57871af3dd46dcfe raw_data/dev/newsdiscusstest2015-fren-ref.en.sgm
18cc8a22b83b6fec68fe3283e8a2ef3e raw_data/dev/newsdiscusstest2015-fren-src.fr.sgm
cf705039c93cda9a9f433fffe376cb34 raw_data/dev/newssyscomb2009-ref.cs.sgm
063ac2b01f9639e8557870cee9653c7f raw_data/dev/newssyscomb2009-ref.de.sgm
78e1ae9398be9999c431c436005a1670 raw_data/dev/newssyscomb2009-ref.en.sgm
0d2bca154e433e225613cf5bcc777bff raw_data/dev/newssyscomb2009-ref.es.sgm
ef320296f23773b214dac3b49036c561 raw_data/dev/newssyscomb2009-ref.fr.sgm
e24724599b481ee4231f0681964c7a6b raw_data/dev/newssyscomb2009-ref.hu.sgm
e71d224b851fb77014e22534cfa26a70 raw_data/dev/newssyscomb2009-ref.it.sgm
afbb687ec4b1584f0a85c9092133a161 raw_data/dev/newssyscomb2009-src.cs.sgm
b11fa2f16d70abc5dcfe90b8d0167e61 raw_data/dev/newssyscomb2009-src.de.sgm
2126a71c2c726aec3302418533ab6e7f raw_data/dev/newssyscomb2009-src.en.sgm
3632ca861af5ab02be27aa6436a5a44f raw_data/dev/newssyscomb2009-src.es.sgm
1a52f5ecef88ab4f834027bcf808770e raw_data/dev/newssyscomb2009-src.fr.sgm
7d99d4c4284be0aec304649e831bc49d raw_data/dev/newssyscomb2009-src.hu.sgm
97f5a343952b3370b7c9569ad8035fac raw_data/dev/newssyscomb2009-src.it.sgm
ca72f09204b5a1e7c5b046774394a87e raw_data/dev/newssyscomb2009.cs
1b15e5b748e1fd085dbedf67869c87ef raw_data/dev/newssyscomb2009.de
a0efdc23a268bcc858c528856d85b33a raw_data/dev/newssyscomb2009.en
f238c8a58aa6d8bf65088287664457aa raw_data/dev/newssyscomb2009.es
27dedff510afd58ed76e66e9a725070f raw_data/dev/newssyscomb2009.fr
0e509a8b7c7f3488fb795ca28122fd5e raw_data/dev/newstest2009-ref.cs.sgm
1ac2d0977f3f66f77b0ff6eba4204f8a raw_data/dev/newstest2009-ref.de.sgm
02752b423282def0fde39d8642881474 raw_data/dev/newstest2009-ref.en.sgm
20c09089345db76d1544ad47c6fa7a83 raw_data/dev/newstest2009-ref.es.sgm
17ff6e37c457594cf8692696d428aa8b raw_data/dev/newstest2009-ref.fr.sgm
44f295f05985a748141ec594e4150961 raw_data/dev/newstest2009-ref.hu.sgm
2d8d509b81fb7391898042ce0d98e563 raw_data/dev/newstest2009-ref.it.sgm
8eac2eff3480140fc1a4ad410dce1900 raw_data/dev/newstest2009-src.cs.sgm
18ab956bd1bbfa6b51b462ddfc06662c raw_data/dev/newstest2009-src.de.sgm
d62ac3e6acd9c5d3567328e92013953b raw_data/dev/newstest2009-src.en.sgm
e73d24ad2817e7638102ef596032aecb raw_data/dev/newstest2009-src.es.sgm
c4928fd5607d05ac7f9ce4955ccba5e2 raw_data/dev/newstest2009-src.fr.sgm
27b66908bf26e7e4208e5e1b0b53f99f raw_data/dev/newstest2009-src.hu.sgm
ec688093aa4df5e9e2625f2cd8f1e72c raw_data/dev/newstest2009-src.it.sgm
d62ac3e6acd9c5d3567328e92013953b raw_data/dev/newstest2009-src.xx.sgm
61a59053e968b66d1c7aa0acb8edd8bb raw_data/dev/newstest2009.cs
2c30a44320eed1ab4a882efd02857bdd raw_data/dev/newstest2009.de
ef5a07243f3081d0d5ecd3524b82a4b3 raw_data/dev/newstest2009.en
3067bda4778b2adfa9cab2bc012208fb raw_data/dev/newstest2009.es
44521b48ec378ab93ee9bde2d4891100 raw_data/dev/newstest2009.fr
85449fc49c7c9a8729ab0820c8bdfe21 raw_data/dev/newstest2010-ref.cs.sgm
dcf4690dfaca3e33537fcef4cbddd9fd raw_data/dev/newstest2010-ref.de.sgm
2052be37e7902b5d700ca1c196f44a11 raw_data/dev/newstest2010-ref.en.sgm
1084df7ad0ac842f1f30afdd34865683 raw_data/dev/newstest2010-ref.es.sgm
eb8a41f2e16e0886975b477d6af2a92a raw_data/dev/newstest2010-ref.fr.sgm
57f4e6ce5281e501dd20cc563f2df4df raw_data/dev/newstest2010-src.cs.sgm
f4e25f698ef4f304eba8daa94fe46bfd raw_data/dev/newstest2010-src.de.sgm
032fab9462c87548e36e8a8aeca41744 raw_data/dev/newstest2010-src.en.sgm
2a08ef80f9fbc61ffaf58b91dc08224c raw_data/dev/newstest2010-src.es.sgm
97cf227b62ba94003ccd7c39236ea79e raw_data/dev/newstest2010-src.fr.sgm
44fe0c72ac6008c644191e2dee8915d3 raw_data/dev/newstest2010.cs
aa3d59046db19ddf9997697c68c3c74b raw_data/dev/newstest2010.de
9d23aa9d0be838be2f85a7bff78ce6f0 raw_data/dev/newstest2010.en
0014ee0b25fe7ca304a7e62ff18e3881 raw_data/dev/newstest2010.es
6136fb2113ddfa48039ea7b261ee9c46 raw_data/dev/newstest2010.fr
da81b878fa01de717c2e38dfe74d0662 raw_data/dev/newstest2011-ref.cs.sgm
206b22bf0d2f286bd13e87925ee53c16 raw_data/dev/newstest2011-ref.de.sgm
bafdf8e5de7c243b9d937f405b1e208c raw_data/dev/newstest2011-ref.en.sgm
4b86af5befd5eb1ad3f0b8e51e9a4693 raw_data/dev/newstest2011-ref.es.sgm
72a65f0be1f34a843fa646fcd085e085 raw_data/dev/newstest2011-ref.fr.sgm
74385262f92b7936ec760a2f70b5c2e1 raw_data/dev/newstest2011-src.cs.sgm
c768d2d7c1d59f756b19b0b282458659 raw_data/dev/newstest2011-src.de.sgm
15f66b409b0fbd51e3d266042d57dc47 raw_data/dev/newstest2011-src.en.sgm
8a308fededd132bfc6eb34b7d4d8238d raw_data/dev/newstest2011-src.es.sgm
768beed6d8a4ba3693ed2593ee155efb raw_data/dev/newstest2011-src.fr.sgm
32051fcf0eb1cad6b5351523af90893c raw_data/dev/newstest2011.cs
c09c3e1883b2cd081f50021b69887ce9 raw_data/dev/newstest2011.de
42244c28949590deb74b593f37d6a82c raw_data/dev/newstest2011.en
994033e6b8546c7b4810a36cfc49cbd5 raw_data/dev/newstest2011.es
60772053ba2332b99db7a5bb84ad6c47 raw_data/dev/newstest2011.fr
48a2e70a7882e5977668be7b5ff16122 raw_data/dev/newstest2012-ref.cs.sgm
77debbee91095e88466a2fcbc01ecb10 raw_data/dev/newstest2012-ref.de.sgm
4d5efd442bb2f843523deca1a36aafd1 raw_data/dev/newstest2012-ref.en.sgm
b5b41d0b6ba15dc79810e07647dd58af raw_data/dev/newstest2012-ref.es.sgm
c9e4b546b91414b34196def73e01594e raw_data/dev/newstest2012-ref.fr.sgm
0792ba330cb2aaa6c7060618df309613 raw_data/dev/newstest2012-ref.ru.sgm
649fcfbd8d8b706bd7f75e30d775d647 raw_data/dev/newstest2012-src.cs.sgm
91b8024827c7a18cd3c4d8d5805c6844 raw_data/dev/newstest2012-src.de.sgm
1af3633cab6c85e71afe46d235f8a841 raw_data/dev/newstest2012-src.en.sgm
0de786bf7a22c7ff32db808f2375d26a raw_data/dev/newstest2012-src.es.sgm
026b732bb653053bbc02fdb6244e0662 raw_data/dev/newstest2012-src.fr.sgm
ce8910c192df6c3d52632cd3faf5bcd3 raw_data/dev/newstest2012-src.ru.sgm
03ce56fa8170e61602aaa6fc38c4f2b8 raw_data/dev/newstest2012.cs
14823046a5a438a1d1c2f09d22f38baa raw_data/dev/newstest2012.de
21b17e5a2dd710b6fc6d659d4498cbb4 raw_data/dev/newstest2012.en
7f56040187fc9d4f6dbe927465cbaa2b raw_data/dev/newstest2012.es
12922ad9ffe8c65d3bf1f8076aea19ad raw_data/dev/newstest2012.fr
2dad778e5008d6f0e8607b35ae5f05bc raw_data/dev/newstest2012.ru
f4d4847b15bd85cfc0578850ebd88edb raw_data/dev/newstest2013-ref.cs.sgm
87bb842dc694d65c703d75eaa2ec49e4 raw_data/dev/newstest2013-ref.de.sgm
c6d6bd214bc3de533b619011d6f7f8ce raw_data/dev/newstest2013-ref.en.sgm
e31fd14c8f4b8035376093d1d6126fa9 raw_data/dev/newstest2013-ref.es.sgm
d66bb0403b4e1cb7fb7eb5751206176c raw_data/dev/newstest2013-ref.fr.sgm
579c0d03ea715d4c2c633a561cdf67fc raw_data/dev/newstest2013-ref.ru.sgm
afe7f6893521e199391420ff725ed23d raw_data/dev/newstest2013-src.cs.sgm
4c417abe9ac3183a804f29c285641506 raw_data/dev/newstest2013-src.de.sgm
b86216894fdf76faced26c301143627e raw_data/dev/newstest2013-src.en.sgm
dff4e9a5696833b338b1356c4c0f7f06 raw_data/dev/newstest2013-src.es.sgm
c6b79404ee5f581e579abf654622ad5f raw_data/dev/newstest2013-src.fr.sgm
78cf164d8e24d55e31aea44b550be3d4 raw_data/dev/newstest2013-src.ru.sgm
6a7e7af5a91b49bffcd7749312c029b4 raw_data/dev/newstest2013.cs
872891e01855b43f2a87155e0c1d2076 raw_data/dev/newstest2013.de
23be23d7b9bd5ad1f2ebd076484332c9 raw_data/dev/newstest2013.en
2bc840a2c850d883532d66c67b4f25ee raw_data/dev/newstest2013.es
77db6a3d2259d00b4d5f45009a3ed375 raw_data/dev/newstest2013.fr
1aed45941a2a4fca5d7c54d4ca63f67b raw_data/dev/newstest2013.ru
0f6923bec65877e6395f73cc2565b596 raw_data/dev/newstest2014-csen-ref.cs.sgm
cf965a55c0f46b6822b424b21af4eb2f raw_data/dev/newstest2014-csen-ref.en.sgm
2ece2d0b521e33618afff968a49d4d4b raw_data/dev/newstest2014-csen-src.cs.sgm
393cffbc087377c98035ccd73c8a8a7a raw_data/dev/newstest2014-csen-src.en.sgm
a05cee5504baa520c1a63b104bb56c87 raw_data/dev/newstest2014-deen-ref.de.sgm
7f907030087b88aeae0303f11378afa4 raw_data/dev/newstest2014-deen-ref.en.sgm
7183d7875ea7ba4a12731140acc8cc8f raw_data/dev/newstest2014-deen-src.de.sgm
5bc31d8d48424640197b89e00cedabb2 raw_data/dev/newstest2014-deen-src.en.sgm
b1dfacd377a34f4f9927dd0947783a1e raw_data/dev/newstest2014-fren-ref.en.sgm
4101e1a3ff6c5ad23aa2f6e0ce0f22bb raw_data/dev/newstest2014-fren-ref.fr.sgm
7b92b0c92b4b7a077bf4cc055142231d raw_data/dev/newstest2014-fren-src.en.sgm
cfd865f26bae037d2db50514d668b195 raw_data/dev/newstest2014-fren-src.fr.sgm
bbe5c5d92bae574967ebcd3af0e658ba raw_data/dev/newstest2014-hien-ref.en.sgm
d04f76b6bf57ec79970cf0f54906573d raw_data/dev/newstest2014-hien-ref.hi.sgm
10ca55c71784fb4ef8f1c612a8e336f0 raw_data/dev/newstest2014-hien-src.en.sgm
1a5f7438ad30f80942f07085225c7dfe raw_data/dev/newstest2014-hien-src.hi.sgm
f97b79ab2a9723be1afcbcadbabb491e raw_data/dev/newstest2014-ruen-ref.en.sgm
4ea80825f9fe641a4327e4c492d253c1 raw_data/dev/newstest2014-ruen-ref.ru.sgm
82f5d4f204485c72203512c9d814170a raw_data/dev/newstest2014-ruen-src.en.sgm
1f22dccd51da1f9163508213a183b45e raw_data/dev/newstest2014-ruen-src.ru.sgm
e7d924c9ca7baa23baae68e2cee7cd54 raw_data/dev/newstest2015-csen-ref.en.sgm
9a29cac72da00f718f88eda30c4ac278 raw_data/dev/newstest2015-csen-src.cs.sgm
8fb26a529c52fbabab17222e97b612d1 raw_data/dev/newstest2015-deen-ref.en.sgm
0dfca16a9c9ad3aea12fdb5c1085efab raw_data/dev/newstest2015-deen-src.de.sgm
0238bdf703da9d81d75f5cba3bcecd0d raw_data/dev/newstest2015-encs-ref.cs.sgm
aa067ef9b7fac031e18d6df357e576d5 raw_data/dev/newstest2015-encs-src.en.sgm
d0cf8dfcd376b2c9136eaba9944bd1a8 raw_data/dev/newstest2015-ende-ref.de.sgm
ade0000673b5a2f0229ec34b3befa950 raw_data/dev/newstest2015-ende-src.en.sgm
bd5b5bbeee6e58ecc2477a3cadebb10b raw_data/dev/newstest2015-enfi-ref.fi.sgm
7e3e983a139c570b776dd0734dfb40cf raw_data/dev/newstest2015-enfi-src.en.sgm
9048de63ac7ceb0ba7ba000babc0cde9 raw_data/dev/newstest2015-enru-ref.ru.sgm
a49f5622aba3461351958a55582da029 raw_data/dev/newstest2015-enru-src.en.sgm
92ad86105ecf7faed99ed97a99b13c9f raw_data/dev/newstest2015-fien-ref.en.sgm
d6526c035382f5d0e4a0c967c1572ab9 raw_data/dev/newstest2015-fien-src.fi.sgm
71428fcc33f147d7a7ee51beb49d6d6a raw_data/dev/newstest2015-ruen-ref.en.sgm
e99f900c5cdaacb38e6a8bd55e8030d4 raw_data/dev/newstest2015-ruen-src.ru.sgm
07d4ee9bb1af67805610ab4731c2aafa raw_data/dev/newstest2016-csen-ref.en.sgm
8f20636841df645b7bb400f7929f5c46 raw_data/dev/newstest2016-csen-src.cs.sgm
3a34a87b0305e3813801fbc097f1dd7d raw_data/dev/newstest2016-deen-ref.en.sgm
23d3fc0b52b863d8ac850d16de2d4794 raw_data/dev/newstest2016-deen-src.de.sgm
092a21aae1a69118d9a0799159f8b72f raw_data/dev/newstest2016-encs-ref.cs.sgm
9fb44e0a85f30bcdc6f779532538643e raw_data/dev/newstest2016-encs-src.en.sgm
381626ee8249023895c30610bbce456b raw_data/dev/newstest2016-ende-ref.de.sgm
2677196bd9afddd6b01b4a6785ec67e1 raw_data/dev/newstest2016-ende-src.en.sgm
aa340b116fa0fa3aa83af2b519796b74 raw_data/dev/newstest2016-enfi-ref.fi.sgm
742fb7aa120742aaf1aef9b1f59356d2 raw_data/dev/newstest2016-enfi-src.en.sgm
e0498e26140a0c695515af473fcf72e0 raw_data/dev/newstest2016-enro-ref.ro.sgm
77f598ea5d0b97c053e6b3130f19a2a1 raw_data/dev/newstest2016-enro-src.en.sgm
8efdb854c94d7d87587a65f93be66ce8 raw_data/dev/newstest2016-enru-ref.ru.sgm
ff756d95230800aa7d06f56dcf08a3dc raw_data/dev/newstest2016-enru-src.en.sgm
2a74a2919942b5ddca4e1e7743aea147 raw_data/dev/newstest2016-entr-ref.tr.sgm
a966abbbeb554b47429cdff61f3d56d5 raw_data/dev/newstest2016-entr-src.en.sgm
8cdb386191e814ae3cc0215da6822747 raw_data/dev/newstest2016-fien-ref.en.sgm
b061d885ee902680edafe421afa16e9b raw_data/dev/newstest2016-fien-src.fi.sgm
8ebbda7d07414c791d4c0bb5897f5adc raw_data/dev/newstest2016-roen-ref.en.sgm
59593b5ce866ae5483e6eaadf762033d raw_data/dev/newstest2016-roen-src.ro.sgm
1e0163031a77e40b6fec8b7dd8288fc0 raw_data/dev/newstest2016-ruen-ref.en.sgm
55e2ea602accd822369434eac47625ba raw_data/dev/newstest2016-ruen-src.ru.sgm
ab9942174352e2498a522865cb536db2 raw_data/dev/newstest2016-tren-ref.en.sgm
660635a258aac67df902d4f7d4865f0e raw_data/dev/newstest2016-tren-src.tr.sgm
ff85f980e33deddca6902bb8034673e4 raw_data/dev/newstestB2016-enfi-ref.fi.sgm
66993925bb1dac7915a93066892f6bb4 raw_data/dev/newstestB2016-enfi-src.en.sgm
7e0acbe86b0d7816300e14650f5b2bd4 raw_data/training-parallel-commoncrawl.tgz
c52404583294a1b609e56d45b2ed06f5 raw_data/training-parallel-europarl-v7.tgz
fc6b83b809347e64f511d291e4bc8731 raw_data/training-parallel-nc-v12.tgz
b4ea563c1b08f6f01689d3cb9948be54 raw_data/training/europarl-v7.cs-en.cs
ce2d7fa76e871288ffa2af5985424b43 raw_data/training/europarl-v7.cs-en.en
f77dd3ab69ea7cc7ef9248e4d96449e1 raw_data/training/europarl-v7.de-en.de
56a330fd9a6291c2540ee733e40e9dd2 raw_data/training/europarl-v7.de-en.en
df1dbeda8edd3838d475dfab242595b3 raw_data/training/europarl-v7.es-en.en
17d03c84750988fdf84705ef4935db7f raw_data/training/europarl-v7.es-en.es
ee3e98f101300baccd2b46bb2592b66e raw_data/training/europarl-v7.fr-en.en
fb585c23a9b38555b8a3f1141d73c5f2 raw_data/training/europarl-v7.fr-en.fr
80bd3d7ab52431574dcf42b8cbc0662c raw_data/training/news-commentary-v12.cs-en.cs
e794ce48b4dfa1f467714a18f99e6bc9 raw_data/training/news-commentary-v12.cs-en.en
2af641b908b038ef198f7e42503c4487 raw_data/training/news-commentary-v12.de-en.de
3ccbc53e3d65efe8db528e7550d00e98 raw_data/training/news-commentary-v12.de-en.en
e08f590c0687fbc2c6d6ab3f80aa670f raw_data/training/news-commentary-v12.es-en.en
b815e8154b7ac3cc7314eb7bd72c6415 raw_data/training/news-commentary-v12.es-en.es
3a4403b8de5822e69cb00a02674feb11 raw_data/training/news-commentary-v12.fr-en.en
65e171cd3810aaf05ee9ce22cf729c50 raw_data/training/news-commentary-v12.fr-en.fr
8c2761ddf974ed1fc1ef26cc83d3999d raw_data/training/news-commentary-v12.ru-en.en
d727222cee01d49e5dc7c5acd223f4c6 raw_data/training/news-commentary-v12.ru-en.ru
09270cbb14bcdfc57d2e9dbc678418f1 raw_data/training/news-commentary-v12.zh-en.en
91ad2ff8e9b9a9c7eccd4faec85b239e raw_data/training/news-commentary-v12.zh-en.zh

Not sure where I'm going wrong

[shengyushen]

Experiencing a verification issue. verify_dataset.sh correctly verifies the md5 sums for newstest2014.en and newstest2014.de but fails to verify the md5 sum for the raw_data directory.

OK: correct tensorflow/newstest2014.en
OK: correct tensorflow/newstest2014.de
ERROR: incorrect raw_data/
ERROR: expected 8fd41a5c658948dfbc1ec83751d9c7fe  -
ERROR: found b21fdd71ac1056a4be466617501bdecf  -

Here is the output of find raw_data/ -type f -exec md5sum {} \; | sort -k 2

c8685312058e23cf08f46abe444a332f  raw_data/commoncrawl.cs-en.annotation
9099f254791097f42952e10f20087cf5  raw_data/commoncrawl.cs-en.cs
480e7d6c2a040ce13fe835a7a0f0017a  raw_data/commoncrawl.cs-en.en
37b003a69178ec22cb3d74a8d9bfb7cc  raw_data/commoncrawl.de-en.annotation
ac590e7b7a1d04ea67b80a6056738dbc  raw_data/commoncrawl.de-en.de
e3d852042ff03ea0ea98d828b98b6c2b  raw_data/commoncrawl.de-en.en
edfa5b076cc2612ddcba84e78096667a  raw_data/commoncrawl.es-en.annotation
34e50983aee1342fcc6ce3cdca085876  raw_data/commoncrawl.es-en.en
f413a156a8a77ffa9ee9d15db338dd12  raw_data/commoncrawl.es-en.es
ed322913456bd3c8288eefc8658c0039  raw_data/commoncrawl.fr-en.annotation
6ac43d37120ebd9815b8b020fc9c479d  raw_data/commoncrawl.fr-en.en
e2c8b6d4ef2c70eb4d0db0c0fc4cc764  raw_data/commoncrawl.fr-en.fr
9598aaa8d74ec7f6265182501e40d05c  raw_data/commoncrawl.ru-en.annotation
4d7b9868a9f0e47c55161b809ed67071  raw_data/commoncrawl.ru-en.en
2af57bbc46a35996c35a728704bb73ce  raw_data/commoncrawl.ru-en.ru
9dc5dd380525793b8e143a3f0dd25ce7  raw_data/dev/newsdev2014.en
f410731105f7cd313148069f096e34c3  raw_data/dev/newsdev2014.hi
042471a75ba56278776ca74e54d167f2  raw_data/dev/newsdev2014-ref.en.sgm
ec3458c0e785dee77fadcd9904fe3941  raw_data/dev/.newsdev2014-ref.en.sgm.swp
0e4a581b13f190c280e4f05985813482  raw_data/dev/newsdev2014-ref.hi.sgm
4c5248f9c7462945c38e857aeb1d6170  raw_data/dev/newsdev2014-src.en.sgm
04144c2031834adcfaf769ce4f18eaaa  raw_data/dev/newsdev2014-src.hi.sgm
b8d952eff69c69ddcde8ad4d0fcd3d57  raw_data/dev/newsdev2015-enfi-ref.fi.sgm
6f04d7c577710c309630f5e95e46eab5  raw_data/dev/newsdev2015-enfi-src.en.sgm
af8109c2ddd33ee2a9f55a21acc4c739  raw_data/dev/newsdev2015-fien-ref.en.sgm
775365f4fe63a7fc09324bfc66bf0428  raw_data/dev/newsdev2015-fien-src.fi.sgm
bc008c5abd9460272633133f141eaf2e  raw_data/dev/newsdev2016-enro-ref.ro.sgm
ed83751e0e925f26a72a7874664be4c0  raw_data/dev/newsdev2016-enro-src.en.sgm
3022225d15ac3688b8166aa8dd236413  raw_data/dev/newsdev2016-entr-ref.tr.sgm
03af6a305a2b90b97364a6032c2b4242  raw_data/dev/newsdev2016-entr-src.en.sgm
1b5b8dedd023369daad6d384048df653  raw_data/dev/newsdev2016-roen-ref.en.sgm
5405823055b9bf13174bbddbe1fd9a41  raw_data/dev/newsdev2016-roen-src.ro.sgm
2685357bce4cae487245e336a832af2d  raw_data/dev/newsdev2016-tren-ref.en.sgm
d0ce333bc57400b03a86e525fe1f5a85  raw_data/dev/newsdev2016-tren-src.tr.sgm
5d7431cfdc0db5a5442c063b6fc96108  raw_data/dev/newsdev2017-enlv-ref.lv.sgm
bb11d739392638db0a35aa376718cda1  raw_data/dev/newsdev2017-enlv-src.en.sgm
e8eee5ded310f6b4cf1c8056a472b455  raw_data/dev/newsdev2017-enzh-ref.zh.sgm
22a0f21e8ddfec362c1fb15adcb957d4  raw_data/dev/newsdev2017-enzh-src.en.sgm
1df951c6a6cf67f45668eff6b863c086  raw_data/dev/newsdev2017-lven-ref.en.sgm
4b422caddea73dad2dd345083bd7c18e  raw_data/dev/newsdev2017-lven-src.lv.sgm
f6e050792e388cb3215e193ca1c8c7c1  raw_data/dev/newsdev2017-zhen-ref.en.sgm
c226046ad0365e3f5b593a5f08b6465a  raw_data/dev/newsdev2017-zhen-src.zh.sgm
a946a6b7b0fc5f63970d7ba2b8196c7b  raw_data/dev/newsdiscussdev2015-enfr-ref.fr.sgm
cd1b574185d964ddbe58d6f4b6a30b95  raw_data/dev/newsdiscussdev2015-enfr-src.en.sgm
05cb1f18c1f811e500c62dde2ed89da4  raw_data/dev/newsdiscussdev2015-fren-ref.en.sgm
90a841d07167f718dd42a9f324eeb940  raw_data/dev/newsdiscussdev2015-fren-src.fr.sgm
51958eb354f2d1dd7621b3988b52359f  raw_data/dev/newsdiscusstest2015-enfr-ref.fr.sgm
05bf9e40ae27a8e9a6c40370ca0fdb53  raw_data/dev/newsdiscusstest2015-enfr-src.en.sgm
d3492954af94acae57871af3dd46dcfe  raw_data/dev/newsdiscusstest2015-fren-ref.en.sgm
18cc8a22b83b6fec68fe3283e8a2ef3e  raw_data/dev/newsdiscusstest2015-fren-src.fr.sgm
ca72f09204b5a1e7c5b046774394a87e  raw_data/dev/newssyscomb2009.cs
1b15e5b748e1fd085dbedf67869c87ef  raw_data/dev/newssyscomb2009.de
a0efdc23a268bcc858c528856d85b33a  raw_data/dev/newssyscomb2009.en
f238c8a58aa6d8bf65088287664457aa  raw_data/dev/newssyscomb2009.es
27dedff510afd58ed76e66e9a725070f  raw_data/dev/newssyscomb2009.fr
cf705039c93cda9a9f433fffe376cb34  raw_data/dev/newssyscomb2009-ref.cs.sgm
063ac2b01f9639e8557870cee9653c7f  raw_data/dev/newssyscomb2009-ref.de.sgm
78e1ae9398be9999c431c436005a1670  raw_data/dev/newssyscomb2009-ref.en.sgm
0d2bca154e433e225613cf5bcc777bff  raw_data/dev/newssyscomb2009-ref.es.sgm
ef320296f23773b214dac3b49036c561  raw_data/dev/newssyscomb2009-ref.fr.sgm
e24724599b481ee4231f0681964c7a6b  raw_data/dev/newssyscomb2009-ref.hu.sgm
e71d224b851fb77014e22534cfa26a70  raw_data/dev/newssyscomb2009-ref.it.sgm
afbb687ec4b1584f0a85c9092133a161  raw_data/dev/newssyscomb2009-src.cs.sgm
b11fa2f16d70abc5dcfe90b8d0167e61  raw_data/dev/newssyscomb2009-src.de.sgm
2126a71c2c726aec3302418533ab6e7f  raw_data/dev/newssyscomb2009-src.en.sgm
3632ca861af5ab02be27aa6436a5a44f  raw_data/dev/newssyscomb2009-src.es.sgm
1a52f5ecef88ab4f834027bcf808770e  raw_data/dev/newssyscomb2009-src.fr.sgm
7d99d4c4284be0aec304649e831bc49d  raw_data/dev/newssyscomb2009-src.hu.sgm
97f5a343952b3370b7c9569ad8035fac  raw_data/dev/newssyscomb2009-src.it.sgm
04b629a9593caa55cb91f60b8ff1e65c  raw_data/dev/news-test2008.cs
f9d7757d029d05d513e81c4ba42d3e03  raw_data/dev/news-test2008.de
49669d8139206c44b6d5daaedf96f219  raw_data/dev/news-test2008.en
8e39422f7e6eed2862e7c442fe279e0d  raw_data/dev/news-test2008.es
d4bb111afbd15564b0e9611783bbee8e  raw_data/dev/news-test2008.fr
73d58db5f47e4e2c5ca1af275ad8805d  raw_data/dev/news-test2008-ref.cs.sgm
8c8f84de9ac3ff60553433b483562fb3  raw_data/dev/news-test2008-ref.de.sgm
ab6551704a20689039db8472b88cc5ea  raw_data/dev/news-test2008-ref.en.sgm
f1c9b0ed953233ff3a96dca014258748  raw_data/dev/news-test2008-ref.es.sgm
02a31f4e26a1b7a4987998f08eb13aad  raw_data/dev/news-test2008-ref.fr.sgm
8e5d54cd743a0d430c7366c86fd55d6f  raw_data/dev/news-test2008-ref.hu.sgm
a2d904389c4bc3a43bce92f7eea9a0e9  raw_data/dev/news-test2008-src.cs.sgm
cb85b28178e8ae6ff1ece09c0628432b  raw_data/dev/news-test2008-src.de.sgm
64e5fb7994b49390aa11f34b222dc870  raw_data/dev/news-test2008-src.en.sgm
2ade1a853d72bc39c12efae03c5a9dd1  raw_data/dev/news-test2008-src.es.sgm
6dbcedd2d42114fac5fc9194c406403b  raw_data/dev/news-test2008-src.fr.sgm
4d83b15d675e0eb85a91b12d9fba856f  raw_data/dev/news-test2008-src.hu.sgm
61a59053e968b66d1c7aa0acb8edd8bb  raw_data/dev/newstest2009.cs
2c30a44320eed1ab4a882efd02857bdd  raw_data/dev/newstest2009.de
ef5a07243f3081d0d5ecd3524b82a4b3  raw_data/dev/newstest2009.en
3067bda4778b2adfa9cab2bc012208fb  raw_data/dev/newstest2009.es
44521b48ec378ab93ee9bde2d4891100  raw_data/dev/newstest2009.fr
0e509a8b7c7f3488fb795ca28122fd5e  raw_data/dev/newstest2009-ref.cs.sgm
1ac2d0977f3f66f77b0ff6eba4204f8a  raw_data/dev/newstest2009-ref.de.sgm
02752b423282def0fde39d8642881474  raw_data/dev/newstest2009-ref.en.sgm
20c09089345db76d1544ad47c6fa7a83  raw_data/dev/newstest2009-ref.es.sgm
17ff6e37c457594cf8692696d428aa8b  raw_data/dev/newstest2009-ref.fr.sgm
44f295f05985a748141ec594e4150961  raw_data/dev/newstest2009-ref.hu.sgm
2d8d509b81fb7391898042ce0d98e563  raw_data/dev/newstest2009-ref.it.sgm
8eac2eff3480140fc1a4ad410dce1900  raw_data/dev/newstest2009-src.cs.sgm
18ab956bd1bbfa6b51b462ddfc06662c  raw_data/dev/newstest2009-src.de.sgm
d62ac3e6acd9c5d3567328e92013953b  raw_data/dev/newstest2009-src.en.sgm
e73d24ad2817e7638102ef596032aecb  raw_data/dev/newstest2009-src.es.sgm
c4928fd5607d05ac7f9ce4955ccba5e2  raw_data/dev/newstest2009-src.fr.sgm
27b66908bf26e7e4208e5e1b0b53f99f  raw_data/dev/newstest2009-src.hu.sgm
ec688093aa4df5e9e2625f2cd8f1e72c  raw_data/dev/newstest2009-src.it.sgm
d62ac3e6acd9c5d3567328e92013953b  raw_data/dev/newstest2009-src.xx.sgm
44fe0c72ac6008c644191e2dee8915d3  raw_data/dev/newstest2010.cs
aa3d59046db19ddf9997697c68c3c74b  raw_data/dev/newstest2010.de
9d23aa9d0be838be2f85a7bff78ce6f0  raw_data/dev/newstest2010.en
0014ee0b25fe7ca304a7e62ff18e3881  raw_data/dev/newstest2010.es
6136fb2113ddfa48039ea7b261ee9c46  raw_data/dev/newstest2010.fr
85449fc49c7c9a8729ab0820c8bdfe21  raw_data/dev/newstest2010-ref.cs.sgm
dcf4690dfaca3e33537fcef4cbddd9fd  raw_data/dev/newstest2010-ref.de.sgm
2052be37e7902b5d700ca1c196f44a11  raw_data/dev/newstest2010-ref.en.sgm
1084df7ad0ac842f1f30afdd34865683  raw_data/dev/newstest2010-ref.es.sgm
eb8a41f2e16e0886975b477d6af2a92a  raw_data/dev/newstest2010-ref.fr.sgm
57f4e6ce5281e501dd20cc563f2df4df  raw_data/dev/newstest2010-src.cs.sgm
f4e25f698ef4f304eba8daa94fe46bfd  raw_data/dev/newstest2010-src.de.sgm
032fab9462c87548e36e8a8aeca41744  raw_data/dev/newstest2010-src.en.sgm
2a08ef80f9fbc61ffaf58b91dc08224c  raw_data/dev/newstest2010-src.es.sgm
97cf227b62ba94003ccd7c39236ea79e  raw_data/dev/newstest2010-src.fr.sgm
32051fcf0eb1cad6b5351523af90893c  raw_data/dev/newstest2011.cs
c09c3e1883b2cd081f50021b69887ce9  raw_data/dev/newstest2011.de
42244c28949590deb74b593f37d6a82c  raw_data/dev/newstest2011.en
994033e6b8546c7b4810a36cfc49cbd5  raw_data/dev/newstest2011.es
60772053ba2332b99db7a5bb84ad6c47  raw_data/dev/newstest2011.fr
da81b878fa01de717c2e38dfe74d0662  raw_data/dev/newstest2011-ref.cs.sgm
206b22bf0d2f286bd13e87925ee53c16  raw_data/dev/newstest2011-ref.de.sgm
bafdf8e5de7c243b9d937f405b1e208c  raw_data/dev/newstest2011-ref.en.sgm
4b86af5befd5eb1ad3f0b8e51e9a4693  raw_data/dev/newstest2011-ref.es.sgm
72a65f0be1f34a843fa646fcd085e085  raw_data/dev/newstest2011-ref.fr.sgm
74385262f92b7936ec760a2f70b5c2e1  raw_data/dev/newstest2011-src.cs.sgm
c768d2d7c1d59f756b19b0b282458659  raw_data/dev/newstest2011-src.de.sgm
15f66b409b0fbd51e3d266042d57dc47  raw_data/dev/newstest2011-src.en.sgm
8a308fededd132bfc6eb34b7d4d8238d  raw_data/dev/newstest2011-src.es.sgm
768beed6d8a4ba3693ed2593ee155efb  raw_data/dev/newstest2011-src.fr.sgm
03ce56fa8170e61602aaa6fc38c4f2b8  raw_data/dev/newstest2012.cs
14823046a5a438a1d1c2f09d22f38baa  raw_data/dev/newstest2012.de
21b17e5a2dd710b6fc6d659d4498cbb4  raw_data/dev/newstest2012.en
7f56040187fc9d4f6dbe927465cbaa2b  raw_data/dev/newstest2012.es
12922ad9ffe8c65d3bf1f8076aea19ad  raw_data/dev/newstest2012.fr
48a2e70a7882e5977668be7b5ff16122  raw_data/dev/newstest2012-ref.cs.sgm
77debbee91095e88466a2fcbc01ecb10  raw_data/dev/newstest2012-ref.de.sgm
4d5efd442bb2f843523deca1a36aafd1  raw_data/dev/newstest2012-ref.en.sgm
b5b41d0b6ba15dc79810e07647dd58af  raw_data/dev/newstest2012-ref.es.sgm
c9e4b546b91414b34196def73e01594e  raw_data/dev/newstest2012-ref.fr.sgm
0792ba330cb2aaa6c7060618df309613  raw_data/dev/newstest2012-ref.ru.sgm
2dad778e5008d6f0e8607b35ae5f05bc  raw_data/dev/newstest2012.ru
649fcfbd8d8b706bd7f75e30d775d647  raw_data/dev/newstest2012-src.cs.sgm
91b8024827c7a18cd3c4d8d5805c6844  raw_data/dev/newstest2012-src.de.sgm
1af3633cab6c85e71afe46d235f8a841  raw_data/dev/newstest2012-src.en.sgm
0de786bf7a22c7ff32db808f2375d26a  raw_data/dev/newstest2012-src.es.sgm
026b732bb653053bbc02fdb6244e0662  raw_data/dev/newstest2012-src.fr.sgm
ce8910c192df6c3d52632cd3faf5bcd3  raw_data/dev/newstest2012-src.ru.sgm
6a7e7af5a91b49bffcd7749312c029b4  raw_data/dev/newstest2013.cs
872891e01855b43f2a87155e0c1d2076  raw_data/dev/newstest2013.de
23be23d7b9bd5ad1f2ebd076484332c9  raw_data/dev/newstest2013.en
2bc840a2c850d883532d66c67b4f25ee  raw_data/dev/newstest2013.es
77db6a3d2259d00b4d5f45009a3ed375  raw_data/dev/newstest2013.fr
f4d4847b15bd85cfc0578850ebd88edb  raw_data/dev/newstest2013-ref.cs.sgm
87bb842dc694d65c703d75eaa2ec49e4  raw_data/dev/newstest2013-ref.de.sgm
c6d6bd214bc3de533b619011d6f7f8ce  raw_data/dev/newstest2013-ref.en.sgm
a72c9746b673fe4bb43caca6f6c42fa2  raw_data/dev/.newstest2013-ref.en.sgm.swp
e31fd14c8f4b8035376093d1d6126fa9  raw_data/dev/newstest2013-ref.es.sgm
d66bb0403b4e1cb7fb7eb5751206176c  raw_data/dev/newstest2013-ref.fr.sgm
579c0d03ea715d4c2c633a561cdf67fc  raw_data/dev/newstest2013-ref.ru.sgm
1aed45941a2a4fca5d7c54d4ca63f67b  raw_data/dev/newstest2013.ru
afe7f6893521e199391420ff725ed23d  raw_data/dev/newstest2013-src.cs.sgm
4c417abe9ac3183a804f29c285641506  raw_data/dev/newstest2013-src.de.sgm
b86216894fdf76faced26c301143627e  raw_data/dev/newstest2013-src.en.sgm
dff4e9a5696833b338b1356c4c0f7f06  raw_data/dev/newstest2013-src.es.sgm
c6b79404ee5f581e579abf654622ad5f  raw_data/dev/newstest2013-src.fr.sgm
78cf164d8e24d55e31aea44b550be3d4  raw_data/dev/newstest2013-src.ru.sgm
0f6923bec65877e6395f73cc2565b596  raw_data/dev/newstest2014-csen-ref.cs.sgm
cf965a55c0f46b6822b424b21af4eb2f  raw_data/dev/newstest2014-csen-ref.en.sgm
2ece2d0b521e33618afff968a49d4d4b  raw_data/dev/newstest2014-csen-src.cs.sgm
393cffbc087377c98035ccd73c8a8a7a  raw_data/dev/newstest2014-csen-src.en.sgm
a05cee5504baa520c1a63b104bb56c87  raw_data/dev/newstest2014-deen-ref.de.sgm
7f907030087b88aeae0303f11378afa4  raw_data/dev/newstest2014-deen-ref.en.sgm
7183d7875ea7ba4a12731140acc8cc8f  raw_data/dev/newstest2014-deen-src.de.sgm
5bc31d8d48424640197b89e00cedabb2  raw_data/dev/newstest2014-deen-src.en.sgm
b1dfacd377a34f4f9927dd0947783a1e  raw_data/dev/newstest2014-fren-ref.en.sgm
4101e1a3ff6c5ad23aa2f6e0ce0f22bb  raw_data/dev/newstest2014-fren-ref.fr.sgm
7b92b0c92b4b7a077bf4cc055142231d  raw_data/dev/newstest2014-fren-src.en.sgm
cfd865f26bae037d2db50514d668b195  raw_data/dev/newstest2014-fren-src.fr.sgm
bbe5c5d92bae574967ebcd3af0e658ba  raw_data/dev/newstest2014-hien-ref.en.sgm
d04f76b6bf57ec79970cf0f54906573d  raw_data/dev/newstest2014-hien-ref.hi.sgm
10ca55c71784fb4ef8f1c612a8e336f0  raw_data/dev/newstest2014-hien-src.en.sgm
1a5f7438ad30f80942f07085225c7dfe  raw_data/dev/newstest2014-hien-src.hi.sgm
f97b79ab2a9723be1afcbcadbabb491e  raw_data/dev/newstest2014-ruen-ref.en.sgm
4ea80825f9fe641a4327e4c492d253c1  raw_data/dev/newstest2014-ruen-ref.ru.sgm
82f5d4f204485c72203512c9d814170a  raw_data/dev/newstest2014-ruen-src.en.sgm
1f22dccd51da1f9163508213a183b45e  raw_data/dev/newstest2014-ruen-src.ru.sgm
e7d924c9ca7baa23baae68e2cee7cd54  raw_data/dev/newstest2015-csen-ref.en.sgm
9a29cac72da00f718f88eda30c4ac278  raw_data/dev/newstest2015-csen-src.cs.sgm
8fb26a529c52fbabab17222e97b612d1  raw_data/dev/newstest2015-deen-ref.en.sgm
0dfca16a9c9ad3aea12fdb5c1085efab  raw_data/dev/newstest2015-deen-src.de.sgm
0238bdf703da9d81d75f5cba3bcecd0d  raw_data/dev/newstest2015-encs-ref.cs.sgm
aa067ef9b7fac031e18d6df357e576d5  raw_data/dev/newstest2015-encs-src.en.sgm
d0cf8dfcd376b2c9136eaba9944bd1a8  raw_data/dev/newstest2015-ende-ref.de.sgm
ade0000673b5a2f0229ec34b3befa950  raw_data/dev/newstest2015-ende-src.en.sgm
bd5b5bbeee6e58ecc2477a3cadebb10b  raw_data/dev/newstest2015-enfi-ref.fi.sgm
7e3e983a139c570b776dd0734dfb40cf  raw_data/dev/newstest2015-enfi-src.en.sgm
9048de63ac7ceb0ba7ba000babc0cde9  raw_data/dev/newstest2015-enru-ref.ru.sgm
a49f5622aba3461351958a55582da029  raw_data/dev/newstest2015-enru-src.en.sgm
92ad86105ecf7faed99ed97a99b13c9f  raw_data/dev/newstest2015-fien-ref.en.sgm
d6526c035382f5d0e4a0c967c1572ab9  raw_data/dev/newstest2015-fien-src.fi.sgm
71428fcc33f147d7a7ee51beb49d6d6a  raw_data/dev/newstest2015-ruen-ref.en.sgm
e99f900c5cdaacb38e6a8bd55e8030d4  raw_data/dev/newstest2015-ruen-src.ru.sgm
07d4ee9bb1af67805610ab4731c2aafa  raw_data/dev/newstest2016-csen-ref.en.sgm
8f20636841df645b7bb400f7929f5c46  raw_data/dev/newstest2016-csen-src.cs.sgm
3a34a87b0305e3813801fbc097f1dd7d  raw_data/dev/newstest2016-deen-ref.en.sgm
23d3fc0b52b863d8ac850d16de2d4794  raw_data/dev/newstest2016-deen-src.de.sgm
092a21aae1a69118d9a0799159f8b72f  raw_data/dev/newstest2016-encs-ref.cs.sgm
9fb44e0a85f30bcdc6f779532538643e  raw_data/dev/newstest2016-encs-src.en.sgm
381626ee8249023895c30610bbce456b  raw_data/dev/newstest2016-ende-ref.de.sgm
2677196bd9afddd6b01b4a6785ec67e1  raw_data/dev/newstest2016-ende-src.en.sgm
aa340b116fa0fa3aa83af2b519796b74  raw_data/dev/newstest2016-enfi-ref.fi.sgm
742fb7aa120742aaf1aef9b1f59356d2  raw_data/dev/newstest2016-enfi-src.en.sgm
e0498e26140a0c695515af473fcf72e0  raw_data/dev/newstest2016-enro-ref.ro.sgm
77f598ea5d0b97c053e6b3130f19a2a1  raw_data/dev/newstest2016-enro-src.en.sgm
8efdb854c94d7d87587a65f93be66ce8  raw_data/dev/newstest2016-enru-ref.ru.sgm
ff756d95230800aa7d06f56dcf08a3dc  raw_data/dev/newstest2016-enru-src.en.sgm
2a74a2919942b5ddca4e1e7743aea147  raw_data/dev/newstest2016-entr-ref.tr.sgm
a966abbbeb554b47429cdff61f3d56d5  raw_data/dev/newstest2016-entr-src.en.sgm
8cdb386191e814ae3cc0215da6822747  raw_data/dev/newstest2016-fien-ref.en.sgm
b061d885ee902680edafe421afa16e9b  raw_data/dev/newstest2016-fien-src.fi.sgm
8ebbda7d07414c791d4c0bb5897f5adc  raw_data/dev/newstest2016-roen-ref.en.sgm
59593b5ce866ae5483e6eaadf762033d  raw_data/dev/newstest2016-roen-src.ro.sgm
1e0163031a77e40b6fec8b7dd8288fc0  raw_data/dev/newstest2016-ruen-ref.en.sgm
55e2ea602accd822369434eac47625ba  raw_data/dev/newstest2016-ruen-src.ru.sgm
ab9942174352e2498a522865cb536db2  raw_data/dev/newstest2016-tren-ref.en.sgm
660635a258aac67df902d4f7d4865f0e  raw_data/dev/newstest2016-tren-src.tr.sgm
ff85f980e33deddca6902bb8034673e4  raw_data/dev/newstestB2016-enfi-ref.fi.sgm
66993925bb1dac7915a93066892f6bb4  raw_data/dev/newstestB2016-enfi-src.en.sgm
9b1aa63c1cf49dccdd20b962fe313989  raw_data/dev.tgz
b4ea563c1b08f6f01689d3cb9948be54  raw_data/training/europarl-v7.cs-en.cs
ce2d7fa76e871288ffa2af5985424b43  raw_data/training/europarl-v7.cs-en.en
f77dd3ab69ea7cc7ef9248e4d96449e1  raw_data/training/europarl-v7.de-en.de
56a330fd9a6291c2540ee733e40e9dd2  raw_data/training/europarl-v7.de-en.en
df1dbeda8edd3838d475dfab242595b3  raw_data/training/europarl-v7.es-en.en
17d03c84750988fdf84705ef4935db7f  raw_data/training/europarl-v7.es-en.es
ee3e98f101300baccd2b46bb2592b66e  raw_data/training/europarl-v7.fr-en.en
fb585c23a9b38555b8a3f1141d73c5f2  raw_data/training/europarl-v7.fr-en.fr
80bd3d7ab52431574dcf42b8cbc0662c  raw_data/training/news-commentary-v12.cs-en.cs
e794ce48b4dfa1f467714a18f99e6bc9  raw_data/training/news-commentary-v12.cs-en.en
2af641b908b038ef198f7e42503c4487  raw_data/training/news-commentary-v12.de-en.de
3ccbc53e3d65efe8db528e7550d00e98  raw_data/training/news-commentary-v12.de-en.en
e08f590c0687fbc2c6d6ab3f80aa670f  raw_data/training/news-commentary-v12.es-en.en
b815e8154b7ac3cc7314eb7bd72c6415  raw_data/training/news-commentary-v12.es-en.es
3a4403b8de5822e69cb00a02674feb11  raw_data/training/news-commentary-v12.fr-en.en
65e171cd3810aaf05ee9ce22cf729c50  raw_data/training/news-commentary-v12.fr-en.fr
8c2761ddf974ed1fc1ef26cc83d3999d  raw_data/training/news-commentary-v12.ru-en.en
d727222cee01d49e5dc7c5acd223f4c6  raw_data/training/news-commentary-v12.ru-en.ru
09270cbb14bcdfc57d2e9dbc678418f1  raw_data/training/news-commentary-v12.zh-en.en
91ad2ff8e9b9a9c7eccd4faec85b239e  raw_data/training/news-commentary-v12.zh-en.zh
7e0acbe86b0d7816300e14650f5b2bd4  raw_data/training-parallel-commoncrawl.tgz
c52404583294a1b609e56d45b2ed06f5  raw_data/training-parallel-europarl-v7.tgz
fc6b83b809347e64f511d291e4bc8731  raw_data/training-parallel-nc-v12.tgz

I got exactly the same error. I am using tf1.13.1, have you solved this problem?

[johntran-nv]

Closing because GNMT is deprecated from the benchmark suite.

[xingzf11]

The following link cannot be accessed, and I cannot use it to download data. Who knows the reason ？

https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.en?token=ABCvMNAuy84bgb5dC-YxSxNyE6qvuit6ks5bGaqtwA%3D%3D
https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.de?token=ABCvMIGPgU8sEiCTPR4D-yovJ-TooXv2ks5bGau_wA%3D%3D