Problems about the eth dataset file in ETH-UCY dataset

[cocoon2wong]

Hi there!

After the discussion between me and @MeiliMa in cocoon2wong/Vertival#1 we found that there are two main different versions of ETH-eth (also called the ETH-univ) dataset files.
One of them has the frame_interval = 10, just like your uploaded files:

Trajectron-plus-plus/experiments/pedestrians/raw/eth/test/biwi_eth.txt

Lines 1 to 7 in 58b9763

	780 1.0 8.46 3.59
	790 1.0 9.57 3.79
	800 1.0 10.67 3.99
	800 2.0 13.64 5.8
	810 1.0 11.73 4.32
	810 2.0 12.09 5.75
	820 1.0 12.81 4.61

And the other has frame_interval = 6, like the original annotation files downloaded here (official) and here with the frames begin with [780, 786, 792, ...].

It appears that the 10-interval file is obtained by linear interpolating the 6-interval file. However, the description of the data set states that

INFO:
The annotation was done at 2.5 fps, that is with a timestep of 0.4 seconds.  

NOTES:
This sequence was acquired from the top of the ETH main building, Zurich, by Stefano Pellegrini and Andreas Ess in 2009. 

(can be found at https://github.com/crowdbotp/OpenTraj/blob/e7b12a0897e57a94b02a735248145c85d84dc01f/datasets/ETH/seq_eth/info.txt#L2-L7)

In addition to this, the original eth video provided at https://data.vision.ee.ethz.ch/cvl/aem/ewap_dataset_full.tgz seems to have been accelerated. In detail, you can see that the seq_eth.avi has been accelerated, while the seq_hotel.avi has been not. Does this mean that interpolating the 6-interval files may be the wrong process method, thus increasing the prediction duration and making your method significantly less effective on eth? I would like to know what you think about this phenomenon, especially because your method behaves rather abnormally on eth dataset.

Looking forward to your reply! 💗

ConghaoWong

[BorisIvanovic]

This is a good question... I have not done as deep of a dive as you have into the original ETH dataset, instead using preprocessed data from other repositories (in particular, we got the ETH/UCY data from the SGAN repository).

It's very possible that if the dataset is accelerated (forcing our model to effectively predict a longer horizon), then model will perform singularly worse on that scene compared to the other scenes. From an evaluation perspective, I guess it does not matter too much as long as everyone either states that the ETH-univ scene is evaluated over a longer history/future horizon (as a result of the accelerated video) or correctly uses the annotations at normal video speed. It seems that, to now, everyone has been doing the former (including this work).

Having said that, I think you should feel free to use the corrected video annotations/speed if you wish and just state in the paper the difference to prior work (you could even show comparisons on both versions of the data, which I think readers would appreciate).

Thanks!
Boris